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ABSTRACT 


In this climate of declining budgets and resources, models and simulations (M&S) 
have become very beneficial to the U.S. Navy. However, the U.S. Navy’s investment in, 
and use of, M&S for addressing critical operational issues (COIs) within a warship’s 
operational test (OT) program would not be practical unless the particular M&S was 
determined to be a credible representation of that which would be physically tested. 
Commander Operational Test and Evaluation Force (COMOPTEVFOR) is responsible 
for accrediting U.S. Navy M&S that are required to support OT. COMOPTEVFOR has 
developed a VV&A process that is documented in COMOPTEVFORINST 5000.1 A. 
This instruction requires all surface ship acquisition program managers (PMs) to develop 
a Validation and Verification (V&V) plan that would meet COMOPTEVFOR’s 
expectations for likely accreditation. This thesis will identify the extent to which surface 
ship PMs are complying with COMOPTEVFORINST 5000.1 A; why they are not in full 
compliance with the instruction; what incentives would help them comply with the 
instruction; and what improvements can be made to the instruction that would increase 
compliance by the PMs. Finally, this thesis will conclude with recommendations that 
would help increase compliance with the instruction by the PMs. 
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I. INTRODUCTION 


A. BACKGROUND 

The models and simulations (M&S) of surface ship perfonnance developed by the 
U.S. Navy are expected to help acquisition program managers (PMs), resource sponsors, 
and operational testers to quickly and affordably determine whether their new or 
modified warships are suitable and effective against projected threats. However, the U.S. 
Navy’s investment in, and use of, M&S for addressing critical operational issues (COIs) 
within a warship’s operational test (OT) program would not be practical unless the 
particular model or simulation was, on the one hand, determined to be a credible 
representation of that which would be physically tested and, on the other hand, seen to be 
risk-acceptable and cost-effective. 

The U.S. Navy establishes the credibility of the models or simulations it uses 
through accreditation by a competent, recognized authority. Commander Operational 
Test and Evaluation Force (COMOPTEVFOR) accredits the U.S. Navy’s M&S used for 
warship OT through its Verification, Validation, and Accreditation (VV&A) process, as 
documented in the updated COMOPTEVFOR Instruction (COMOPTEVFORINST) 
5000.1 A of 9 September 2004. This instruction requires that all surface ship 1 PMs, who 
plan to develop M&S for OT, run a standardized VV&A process 2 that interacts with 
COMOPTEVFOR - in both its M&S accreditor and operational tester functions - as they 
develop their M&S. 

In practice, however, few PMs have been able to act in full accordance with the 
tenns of the COMOPTEVFOR process - and yet their M&S are eventually accredited, 
though not without a certain amount of contention and political brinksmanship among the 
parties. Thus, the question arises: Why are U.S. Navy surface ship acquisition program 
managers not fully complying with the standardized VV&A process set forth by the 
accreditation authority? Both the material developer and the operational tester endeavor 

1 This includes the weapons systems aboard surface ships. COMOPTEVFOR’s M&S VV&A policy 
applies as well to the U.S. Navy’s other warfighting platforms, e.g., submarines, aircraft. 

2 COMOPTEVFOR’s standardized VV&A process will be described and discussed in Chapter III. 


1 



to demonstrate that warfighter requirements will be met - in this case through M&S - yet 
each seems to be looking differently at the benefits, costs, and risks entailed to achieve 
these ends. Thus, a practical VV&A process would need to strike a realistic balance 
between different, though legitimately held, institutional attitudes. The updated process 
described in COMOPTEVFORINST 5000.1 A should represent progress in managing this 
balance and enabling compliance to achieve the objectives of the regulation, cost- 
effective use of models and simulations. 

B. PURPOSE AND OBJECTIVES 

This thesis endeavors to identify barriers to compliance with the VV&A process 
that limit the ability to achieve cost-effective use of models and simulations for OT. 

The objectives of this thesis are to: (1) identify the extent to which surface ship 
acquisition program managers are complying with the updated VV&A process prescribed 
by COMOPTEVFORINST 5000.1 A; (2) establish why PMs are not fully complying with 
the prescribed VV&A process; (3) determine what factors help PMs fully comply with 
the instruction; and (4) identify what improvements can be made to the VV&A process. 


C. SCOPE 

This thesis will study the VV&A process required by COMOPTEVFORINST 
5000.1 A of 9 September 2004 as it applies to U.S. Navy surface ship (and weapons) 
M&S used for OT. It will assess stakeholder familiarity with the instruction, examine the 
PM’s current level of compliance, and conclude whether that level is acceptable. The 
VV&A experience of recent U.S. Navy surface ship (and weapons) acquisition programs 
will be sampled. At the heart of the research, stakeholder views will be collected on: the 
(1) benefits, (2) costs, and (3) risks seen in compliance; and the (4) benefits, (5) costs, 
and (6) risks seen in non -compliance. Finally, this thesis will consider potential 
improvements to COMOPTEVFOR’s VV&A process that could be embodied in a 
revision to COMOPTEVFORINST 5000.1 A. 
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The scope of this thesis is intentionally narrowed to address the OT of U.S. Navy 
surface ships and their weapons systems. The research herein does not cover M&S used 
for non-OT (e.g., developmental T&E, resource requirement studies, etc.). Neither the 
M&S used for other U.S. Navy warfighting platforms (e.g., aircraft), nor that used for 
other U.S. military (e.g., U.S. Anny), foreign military, or U.S. civilian activities are 
addressed. Furthermore, this thesis emphasizes the accreditation aspect of the VV&A 
process over its verification and validation steps. Nevertheless, the conclusions reached 
by this thesis, while focused on U.S. Navy surface ship (and weapons) programs, are 
intended for general applicability to all weapons systems, platforms, and capabilities 
acquired by the U.S. Department of Defense (DoD) where M&S is used (and 
consequently VV&A is needed) to satisfy OT requirements. 

D. RESEARCH QUESTIONS 

This thesis considers a primary and four subsidiary research questions. 

1. The Primary Research Question Posed By This Thesis Is: 

What are the barriers to compliance with COMOPTEVFOR’s VV&A process that 
U.S. Navy surface ship (and weapons) acquisition programs face? 

2. The Subsidiary Research Questions Are: 

a. What benefits and liabilities are seen by the stakeholders in 
subscribing to the COMOPTEVFORINST 5000.1 A VV&A process, if its standards were 
consistently followed by all U.S. Navy surface ship (and weapons) acquisition programs? 

b. What are the cost and schedule considerations that impact U.S. 
Navy surface ship (and weapons) acquisition programs in the course of complying with 
the COMOPTEVFORINST 5000.1 A VV&A process? 

c. What are the risks observed by the stakeholders when the 
COMOPTEVFORINST 5000.1 A VV&A process is not followed? 
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d. What actions can be taken by stakeholders to ensure compliance 

with COMOPTEVFOR's VV&A process? 

E. METHODOLOGY 

The thesis research and analysis proceeded through seven steps: 

1. Stakeholder Identification 

Major stakeholders, who participate in this arena of the Navy’s VV&A process, 

are identified and grouped by category. The major stakeholder groups that were 
identified are: (a) Tester/Accreditor; (b) Resource Sponsor; (c) Program Manager; and (d) 
VV&A Standards Official. 

2. Literature Review 

Documents that provide information and data on the practices and expectations of 
VV&A are identified and reviewed. 

3. VV&A Forums 

The researchers attended and participated in five forums (i.e., technical working 
group meetings), sponsored by the U.S. Navy’s VV&A standards community, to obtain 
assistance in the identification of active stakeholders, current issues, and relevant 
literature. Forum attendance also provided opportunities to interview stakeholders. 

4. Semi-structured Interviews 

The researchers conducted fourteen semi-structured interviews of participants 
representing the major stakeholder groups for VV&A of M&S used for OT of newly 
acquired (or modified) U.S. Navy surface ships and their weapons systems. 
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5. Data Analysis 

Data from the semi-structured interviews was analyzed and reported for each of 
ten interview questions by VV&A stakeholder category and management level. 


6. Conclusion Development 

Conclusions were developed by a systematic review of the results against the 
primary and subsidiary research questions. The conclusion topics are as follows: 

a. Extent of compliance with COMOPTEVFORINST 5000.1 A 

b. Barriers to compliance with COMOPTEVFORINST 5000.1 A. 

c. Factors for compliance with COMOPTEVFORINST 5000.1 A. 

d. Possible improvements to COMOPTEVFORINST 5000.1 A 


7. Recommendation Formulation 

Recommendations were formed through consideration of the conclusions and the 
researchers’ general experience and expectations for the future of VV&A. Topics for 
further studies are identified. 

F. ORGANIZATION 

Chapter II provides background information on M&S that describes DoD 
strategies and how M&S is used by the U.S. Navy to support surface ship acquisition. 

Chapter III provides background information on VV&A that describes DoD 
processes, COMOPTEVFORINST 5000.1 A, and other VV&A processes used by U.S. 
Navy surface ships program offices. 

Chapter IV describes the research methodology used in this study. 

Chapter V analyzes the results of the data collected from the interviews. 

Chapter VI provides conclusions, which answer the thesis research questions, and 
offers recommendations and topics for further studies. 
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II. BACKGROUND ON MODELING AND SIMULATION 


A. INTRODUCTION 

This chapter discusses the M&S strategies of the U.S. Department of Defense 
(DoD) and the U.S. Department of the Navy (DoN) and provides an overview on the role 
of modeling and simulation (M&S) in addressing the operational test (OT) of U.S. Navy 
surface ships and how M&S is being used to resolve Critical Operational Issues (COIs) 
within OT. 

B. DOD M&S STRATEGY 

1. Introduction 

DoD has directed its Service Components (i.e. the Army, Navy, Marine Corps, 
and Air Force) to use M&S to reduce time, resources, and risks of acquisition, to increase 
the quality of systems acquired, and to improve system performance. [1] In the post Cold 
War era of declining defense budgets, reduced manning and rapidly evolving technology 
and threats, M&S can provide tailored test environments that extend test envelopes to 
augment live testing. [1] These benefits are obtained only if the M&S are credible 
representations of the tested systems, projected threats, and the test environments. [1] 

The widespread use of advanced M&S is necessary if the DoD is to cost- 
effectively maintain readiness, accomplish its operational missions, make optimal 
investment decisions, and achieve dramatic acquisition savings. [2] To this end, DoD has 
established a strategy to foster simulation interoperability and reuse, calling for them to 
work together as a community, in systems of systems. [2] 

In the late 1980 and early 1990s, DoD and Congress began to comprehend the 
vast potential of advanced M&S to revolutionize the way the Department made decisions 
and conducted its operations. [2] DoD has used M&S in many diverse ways through 
most of its history, but advances in information technology capabilities and the advent of 
the distributed interactive simulation concept, evidenced by projects such as Defense 
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Advanced Project Agency’s (DARPA) Simulation Network (SIMNET), clearly signaled 
that major increases in simulation capabilities were possible. [2] 

Encouraged by several Defense Science Board studies pointing to the potential 
benefits, DoD decided it should make the use of advanced simulation a corporate priority. 
[2] In the summer of 1991, the Deputy Secretary of Defense put a single individual, the 
Under Secretary of Defense for Acquisition (now the Under Secretary of Defense for 
Acquisition, Technology and Logistics (USD (AT&L)), in charge of the Department’s 
simulation efforts. [2] At the same time, the USD (AT&L) established a flag-level 
Executive Council on Modeling and Simulation (EXCIMS) and stood up the Defense 
Modeling and Simulation Office (DMSO) as a new organization within the Office of the 
Director of Defense Research and Engineering (DDR&E). [2] DMSO was designated the 
focal point for M&S matters across DoD. This management structure was subsequently 
codified in DoD Directive 5000.59, “DoD Modeling and Simulation (M&S) 
Management,” of January 4, 1994. [2] 

In March 1992, the EXCIMS, chaired by the DDR&E, crafted a vision for DoD 
M&S. [2] This vision called for the establishment of readily available, operationally 
valid synthetic environments to support a full range of application areas (e.g., joint 
training and doctrine development, fonnulation and assessment of operational plans, and 
support of the acquisition process and force structuring). [2] The vision also stated that 
these environments would be constructed from affordable, reusable components that 
interoperate through open systems architecture. [2] 

The vision called for achieving interoperability and M&S reuse as a fundamental 
strategy to satisfy DoD’s simulation needs. Since it was not possible to build just a few 
simulations to satisfy the broad needs of DoD, and because any simulation would be an 
abstraction of the real world driven by the goals of its sponsor, there would need to be 
many different simulations that certainly could not be individually planned and controlled 
by one central office. [2] However, at the other extreme, the cost of building narrowly 
focused, “stove-piped” simulations to satisfy each of the ever-expanding simulation 
requirements would be too high. [2] 
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The DoD M&S community appreciated that cost, time, and risk reduction benefits 
would come from reusing existing simulations to address new requirements and flexibly 
connecting existing representations together to create new simulation environments. [2] 
Demands for the reuse of simulations came soon and frequently, often even before their 
development was complete, but it proved difficult to achieve the necessary 
interoperability. [2] The key technical question was how to facilitate interoperability and 
reuse. [2] The Distributed Interactive Simulation (DIS) and Aggregate Level Simulation 
Protocol (ALSP) interoperability Standards, which had emerged in the early 1990s, 
represented major advances in achieving interoperability, but each applied to only a 
particular application domain, had significant technical limitations, and could not satisfy 
the broad simulation interoperability and reuse goals of the DoD. [2] A new approach 
was required. 

As these lessons were learned, DoD, led by the DDR&E, established a new, more 
comprehensive strategy to foster broad interoperability and reuse. [2] The analogy was 
drawn to planning a city, calling for simulations to work together as a community, in a 
system of systems. [2] To foster a community of simulations capable of being composed 
into federations and addressing whatever requirements emerge, requires a common 
technical framework (CTF) and a set of common services. [2] 

This “city planning” strategy was captured in the first DoD-wide M&S Master 
Plan, DoD 5000.59P, which was signed in October 1995 after formal coordination across 
all DoD Components. [2] The M&S Master Plan (MSMP) identified six objectives 
necessary to achieve DoD’s M&S vision. [2] The first and most important of the 
objectives calls for the establishment of a CTF, to which individual simulation 
developments must conform. [2] With this strategy, DoD expects to foster 
interoperability, reusability, cost-effectiveness, and greater capability. [2] 

2. Common Technical Framework 

The CTF consists of three parts, the most important of which is the high-level 
architecture (HLA). [2] An “architecture” defines the major functional components, 
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design rules, and interfaces for a computer-based simulation system. [2] It specifies 
(conceptually) how they hook together and work together as a whole and is distinct from 
the software that is required to implement it. [2] The other two legs of the CTF are a 
common understanding of the real-world (tenned a conceptual model of the mission 
space, or CMMS) and data Standards. [2] 


a. High Level Architecture 

The HLA has been developed, tested, and established as the standard 
technical architecture for all DoD simulations. [2] DoD policy had required simulations 
to comply with the HLA by 1 October 2000, be retired, or obtain a waiver from the USD 
(AT&L). [2] As of 1 Aug 1997, approximately 403 simulations had been committed to 
HLA compliance. [2] 

The HLA was developed by consortium of major simulation programs. [2] 
It was chartered by EXCIMS, led by DMSO and supported by govermnent, academic, 
and commercial sponsors. [2] In taking this approach, DoD emulated the commercial 
industry Standards development processes, which was a significant innovation for DoD 
and one of the major ingredients for the success of the HLA enterprise. [2] It allowed 
broad community participation in the HLA development process, fostered ownership of 
the HLA, and expanded the technical experience base. [2] 

The HLA is not software, but a set of rules and specifications that 
prescribe how simulations, live systems (e.g., command and control systems, weapon 
systems on instrumented ranges), and supporting utilities such as data collectors and 
wide-area viewers work together as a federation. [2] The HLA separates the 
representations, provided by simulations or live systems, from the general 
interconnection services that is provided by a Runtime Infrastructure (RTI). [2] 

Three things define the HLA: rules, an interface specification, and an 
object model template. [2] The ten HLA Rules establish the requirements for federation 
components (term federates) and the federation as a whole. [2] The HLA Object Model 
Template (OMT) specifies a standard fonn in which federates and federations must be 
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described. [2] Finally, the HLA Interface Specification describes the ways simulations 
interact with the supporting RTI software during a federation operation. [2] 

The HLA is garnering much international and commercial interest. [2] 
The North Atlantic Treaty Organization (NATO) has established a high-level steering 
group on M&S that is evaluating the HLA as a potential NATO Standard to facilitate 
simulation interoperability and reuse. [2] The Simulation Interoperability and Standards 
Organization (SISO) has begun the process of establishing the HLA as an Institute of 
Electrical and Electronics Engineers (IEEE) Standard. [2] The DARPA Synthetic 
Theater of War (STOW), with United Kingdom participation, is using the HLA and 
pushing technology boundaries in a number of important areas, including scalability, 
command and control representation, and representations of the natural environment. [2] 
Many foreign nations have begun HLA-based simulation developments. [2] 

b. Conceptual Models of the Mission Space 

Conceptual Models of the Mission Space (CMMS) is a first abstraction of 
the real world activities associated with a particular mission. [2] Such conceptual models 
provide entities, actions, tasks, and interactions (EATI) representation of the military 
mission space that is independent of any specific computer-aided software engineering 
(CASE) tool or utility employed to capture it. [2] These functional descriptions are 
intended to serve as a representational resource, capturing information about various 
military operations, to be used by simulation developers and others. [2] This information 
is derived from authoritative sources, is described using common syntax and semantics, 
and is independent of any particular simulation implementation. [2] 

Development and sharing of the CMMS is a responsibility shared among 
various organizations in DoD. [2] Warfighters act as the authoritative source for how the 
world works, specifying mission-essential task lists and doctrine. [2] The developers of 
the simulation, cooperatively and collectively, provide the knowledge for acquisition. [2] 
DMSO develops the database management system: (1) providing knowledge acquisition 
teams with technical support (e.g., common semantics and syntax, data interchange 
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formats); (2) registering the resulting CMMS data provided by the simulation developers: 
and (3) allowing wide access to the CMMS database as it is fdled. [2] 

Other related DMSO efforts such as the Modeling and Simulation 
Resource Repository (MSRR) and the Data Standards Work, provide the infrastructure 
and technologies needed to support CMMS. [2] The MSRR provides CMMS access to 
simulation developers, doctrine developers, trainers, and other interested parties. [2] The 
CMMS database management software supports a variety of structured views for the 
display and manipulation of these conceptual models. [2] A Data Interchange Format 
(DIF) Standard has also been developed for the conversion, integration, storage, and 
extraction of these conceptual models. [2] 

c. Data Standards 

DMSO led the effort to develop M&S data Standards as directed by the 
DoD MSMP. [2] DMSO’s data engineering efforts have four thrusts. [2] The first is the 
development of Common Semantics and Syntax (CSS) and associated DIFs. [2] CSS are 
the logical structure and content (meaning) of any specific model or data element. DIFs 
are the physical representation (e.g., Backus-Naur Form, Structured Query Language 
(SQL), bits and bytes) of data that programmers employ to interchange complex data 
fields. [2] 

The second thrust of the data Standards efforts is the identification of 
authoritative data sources (ADS). [2] ADS tells developers and users where to go for the 
best data. Each military service is identifying the appropriate ADS associated with its 
varied responsibilities (e.g., weapon system characteristics and performance, order of 
battle) and providing that information to DMSO for incorporation in the MSRR. [2] 

The third thrust is the establishment of standard Data Quality practices to 
provide a means to ensure databases are complete and coherent. [2] The fourth thrust is 
the establishment of Data Security practices (DS) to guide the sensitive matter of data 
access and release. [2] 
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Data Standards are intended to facilitate the creation, management, and 
exchange of information data sources, simulation developers, and simulation users for all 
phases of simulation operations - development, initialization, runtime interchange, and 
post-execution review. [2] They are employed in the drafting of HLA Simulation Object 
Models (SOMs) and Federation Object Models (FOMs), CMMS, and in the authoritative 
representation of the natural environment, units, systems, and human behavior. [2] 

3. Common Services 

A broad range of common services is being developed to support both users and 
developers of M&S. [2] 

a. Modeling and Simulation Executive Agents 
To provide focus, coordination, centers of excellence, cost avoidance, and 
broad support to the M&S community in common and general use areas, the DoD has 
designated four M&S Executive Agents (MSEAs) to serve the broad M&S community. 
[2] They represent the three domain areas of the natural environment (terrain, oceans, 
aerospace) and a combined domain of foreign forces and joint national intelligence 
processes. [2] 


b. Natural Environments 

The second major objective of the DoD M&S Master Plan (MSMP) is to 
provide timely and authoritative representations of the natural enviromnent, which are 
subdivided into the terrain, oceans, and aerospace domains. [2] 

(1) Terrain. In April 1995, the USD (AT&L) designated the 
Director, Defense Mapping Agency (DMA) as the DoD MSEA for Authoritative 
Representation of the Terrain Natural Environment. [2] 

(2) Oceans. The Secretary of the Navy was designated the MSEA 
for the Ocean Environment in April 1996. 
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(3) Aerospace. The Secretary of the Air Force was designated the 
MSEA for the Air and Space Natural Environments in April 1996. [2] 

c. Foreign Forces and U.S. National & Joint Intelligence Processes 

The Director of the Defense Intelligence Agency (DIA) is the MSEA for 

the Representation of Foreign Forces and U.S. National and Joint Intelligence. [2] 

d. Communication Services 

Most simulation executions take place within a single facility or usually 
involve only limited point-to-point wide area communications. [2] Additionally, these 
distributed simulations may require multi-cast and bandwidth reservation services which 
are not generally available within either commercial or defense communication networks. 
[2] The Defense Infonnation Services Network (DISN), DARPA and the Defense 
Information Systems Agency (DISA) have cooperatively put in place the necessary 
communication capabilities required to support simulation activities that include 
enhanced Internet Protocol (IP) services. [2] 

e. Verification, Validation, and Accreditation (VV&A) 

The DoD’s strategy for VV&A is discussed in Chapter III. 

f Modeling and Simulation Resource Repository 

The Modeling and Simulation Resource Repository (MSRR) is a set of 
resources stored on a distributed network of computers, and linked by special 
applications software and world-wide web (WWW) protocols. [2] The purpose is to 
catalogue information and provide a means for identifying and distributing reusable 
resources. [2] 
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g. Information Resources 

There are two human-in-the-loop information resources, i.e., help desks, 
available to the DoD M&S community. [2] The first is the Defense Modeling Simulation 
and Tactical Technology Infonnation Analysis Center (DMSTTIAC), which is a 
government-funded agency that provides scientific and technical information and 
analysis services. [2] It serves the M&S, special Operations, test and evaluation, and 
tactical warfare communities. [2] 

The second is the Modeling and Simulation Operational Support Activity 
(MSOSA), which is a contractor-staffed activity operating under the direction of DMSO. 
[2] Its mission is to assist DoD activities in meeting their M&S needs by providing 
operational advice and facilitating access to M&S infonnation and assets. [2] 


C. U.S. NAVY M&S STRATEGY 

1. Introduction 

The use of M&S in the U.S. Navy is greater now than ever before. [3] Its 
(simulation) ability to represent a platfonn or process abstractly improves the U.S. 
Navy’s capacity to train its personnel, plan and conduct operations, develop and procure 
weapon systems, and analyze force employment alternatives. [3] 

The major U.S. Navy simulation programs are structured in terms of their areas of 
application, the requirements they address, the capabilities they provide, and the funding 
they receive. [3] Major programs are defined as those being well recognized within each 
community. [3] Simulations are an integral part of analysis, training, education, and 
acquisition processes; and as a technology, the use of M&S is pervasive. [3] In spite of 
these challenges, providing a simulation-specific baseline is critical. First, a baseline 
articulates simulation use and investment that allows U.S. Navy-wide priorities to be 
refined. [3] Next, it describes application and funding trends, enabling U.S. Navy leaders 
to promote needed technology and resource expenditures. [3] Finally, it supports 
simulation planning and documents key trends and issues requiring remedies. [3] 
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2. DoN Modeling and Simulation Vision and Goal 

The U.S. Navy’s vision is that M&S “will be a pervasive tool for operational units 
and will support analysis, training, and acquisition throughout DoN.” [4] The U.S. Navy 
M&S vision calls for the use of world-class models, simulations, and simulators as tools 
to meet the future challenges of supporting force readiness, structure, and transfonnation. 
[4] 

The goal of the U.S. Navy’s M&S community is to provide capable, reactive, and 
flexible simulation systems that are always available to improve operational decision 
making, assessment and experimentation, training, and acquisition. [5] Attainment of 
this goal, which supports the vision, will improve the ability of all U.S. Navy decision 
makers to execute their missions and accomplish their objectives. [5] 

3. DoN M&S Organizations 

a. Navy Modeling and Simulation Office 

SECNAV Instruction 5200.38A, Department of the Navy Modeling and 
Simulation Management, established the Navy Modeling and Simulation Office 
(NMSO), formerly known as the Navy Modeling and Simulation Management Office 
(NAVMSMO), in the Office of the Chief of Naval Operations. [6] NMSO is responsible 
for coordination and integration of M&S efforts across all U.S. Navy warfare and support 
areas. [4] The U.S. Navy uses and relies on M&S in four pillars: assessment, training, 
acquisition, and support to operations. These areas cross organizational boundaries and 
represent the many interests and responsibilities of the U.S. Navy. [4] 

b. U.S. Navy M&S Standards Steering Group (MS3G) 

The MS3G has representation from the U.S. Navy Secretariat Office, the 
Chief of Naval Operations, Fleets, Systems Commands, laboratories, Federally Funded 
Research and Development Centers (FFRDCs), and University Affiliated Research 
Centers (UARCs). [6] This broad participation and team effort ensures that candidate 
Standards are rigorously evaluated and appropriately promoted. [6] Participation in the 
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MS3G is limited to Government personnel and their designated representatives. [6] The 
mission of the MS3G is to oversee the operation of the U.S. Navy M&S Standards 
Process, rather than act as a technical deliberation body. [6] Technical reviews of 
candidate Standards are delegated to specialized subgroups. [6] 

c. Subgroups 

The MS3G is responsible for oversight of all areas of the Navy M&S 
Standards Project. [6] Technical expertise is provided by the M&S Standards Subgroups, 
each focused on a specific area (applications, such as logistics; or technologies, such as 
data or VV&A). [6] These subgroups are permanent in nature; each subgroup is chaired 
by a member of the MS3G and is appointed by the Director, Navy M&S Standards 
Project. [6] Membership in M&S Standards Subgroups is open to representatives from 
government, industry and academia. [6] The M&S Standards Subgroup is responsible for 
a U.S. Navy Standard from the time the need is identified to the time it is approved, 
promulgated and advocated. [6] These are the subgroups: 

(1) Application Planning and Review Groups (APRGs). APRGs 
identify specific application needs for potential U.S. Navy M&S Standards, and represent 
experts in functional areas of the U.S. Navy M&S Community. [6] 

(2) Technology Area Groups (TAGs). TAGs address needs as 
they are identified, in the appropriate area of technology, by researching, analyzing, and 
reviewing M&S Standards submissions. [6] 

(3) Special Interest Groups (SIGs). When the need for a Standard 
is identified and there is no existing APRG or TAG to address this need, a Special 
Interest Group (SIG) will be established by the MS3G Chair. [6] The responsibilities of a 
SIG are the same as that of an APRG or TAG, as detennined by the MS3G Chair. [6] 
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4. 


U.S. Navy M&S Administrative Procedures 


a. MS3G Meetings 

A quorum for the MS3G is defined as a simple majority of the voting 
members of the MS3G (more than one half). [6] If a member organization fails to attend 
two consecutive meetings, the MS3G Chair shall notify the organization that it will lose 
its right to vote and be placed on an inactive membership list unless represented at the 
next meeting. [6] If the organization is not then represented, it shall be considered 
inactive and a nonvoting member. [6] 

b. Voting 

MS3G voting occurs on a nominated Standard through the use of an online 
web system and only in the 5 th Stage of the Navy M&S Standard Process. [6] The voting 
pool is dynamic and unique, as detennined by the MS3G (or the MS3G chair) for 
proposed Standards undergoing review. [6] 

c. Legal Issues 

Because of copyright laws, copyrighted Standards submitted for review 
cannot be posted publicly. [6] Licensed copies of copyrighted material used by members 
of the review team are obtained separately. [6] Proposed Standards not already approved 
by a recognized Standards organization can still be considered as U.S. Navy M&S 
Standards candidates and U.S. Navy M&S users can still use them to meet their needs. 
[6] If a specific Standard identified by the U.S. Navy M&S community is later approved 
as a government or DoD Standard, it will be identified as approved by a different 
authority. [6] 


d. File Retention 

The Standards Coordinator (SC) is responsible for maintaining active and 
archival files for the MS3G meeting proceedings and any other document produced by or 
for the MS3G in the conduct of U.S. Navy M&S Standards activities. [6] These files 
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shall be retained for a minimum of five years and shall include the following: (1) meeting 
announcements; (2) published agendas; (3) meeting minutes; and (4) meeting 
presentations (project and guest briefings). [6] 

5. U.S. Navy M&S Processes for Standards Review and Approval 

The vision of the U.S. Navy M&S Standards Project is to promote a common set 
of Standards (supporting protocols, techniques, best practices, etc.) for the use and reuse 
of U.S. Navy models, simulations, and data. [6] It is not intended to create a set of 
written Standards that remain unused in the U.S. Navy M&S Standards repository. [6] It 
is important to apply a process that permits the U.S. Navy M&S community to keep pace 
with, rather than react to, technological advances. [6] 

The U.S. Navy M&S Standards conceptual process consists of three key 
activities: (1) Nominate, (2) Evaluate, and (3) Advocate U.S. Navy M&S Standards. [6] 
Essential to these key elements are: automated web tools; collaboration and facilitated 
support to address and resolve issues; and M&S experts to review, leverage, and refine 
the appropriate Standards. [6] 

a. Types of Standards 

The U.S. Navy M&S Standards process supports the submission of 
nominated Standards that are recommended in particular technical and application areas 
as well by the type of Standard the submission provides. [6] An online submission form 
is available in the online Standards Nomination, Evaluation, Advocacy and Central 
Repository System (SNEACRS) for submitting a potential U.S. Navy M&S Standard. [6] 
Immediately upon submission, the nomination is assigned a system-generated number for 
tracking purposes. [6] The nomination is now called a Standards Needs Document 
(SND) and is identified throughout the system by this assigned number. [6] 

b. Standards Review and Approval Process Stages 

The execution of the U.S. Navy M&S Standards process requires an 
understanding of the flow of activities, their relationships, dependencies, and associated 
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data. [6] The U.S. Navy M&S Standards process is broken down into eight stages (or 
milestones), which are: [6] 

(1) Initial submission 

(2) Evaluation by MS3G Chair and Primary Subgroup 

(3) Internal review by NMSO and MS3G 

(4) U.S. Navy Community Review 

(5) Determine Disposition - MS3G Vote 

(6) Determine Disposition - Director, NMSO Approval 

(7) Promulgate and advocate Standard 

(8) Periodic review by Primary Subgroup 

6. U.S. Navy M&S Standards Project Charter 

a. Purpose 

The purpose of the U.S. Navy MS3G is to promote a common set of 
Standards for the use and reuse of U.S. Navy models, simulations, and data; as well as 
supporting protocols, techniques and processes. [6] 

b. Authority 

The U.S. Navy MS3G was established under the authority of the Navy 
Modeling and Simulation Management Office (NAVMSMO) (OPNAV N6M) on 8 Nov 
99.[6] 


c. U.S. Navy M&S Standards Project 

The U.S. Navy M&S Standards Project is structured to make every effort 
to ensure that appropriate M&S Standards are available when needed for U.S. Navy 
M&S development, acquisition, deployment, and life-cycle support. [6] It also ensures 
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that the necessary infrastructure is present and applied to the U.S. Navy M&S process. 

[6] Standards endorsed by this process may be promulgated as mandatory or advisory 
(recommended). [6] The Project identifies and establishes a strategy for achieving future 
U.S. Navy M&S Standards evolution. [6] To ensure that appropriate U.S. Navy M&S 
Standards are available when needed by the U.S. Navy M&S community, frequent and 
effective interactions are required among the various participants in the U.S. Navy 
Standards Process. [6] 

d. Participants 

(1) Director, NMSO. Sponsors and provides requirements and 
policy guidance for the project. [6] 

(2) Project Director. Chairs the U.S. Navy MS3G and is 
appointed by the Director, NMSO. [6] 

(3) The U.S. Navy M&S User Community. Produces and utilizes 

M&S Standards. [6] 

(4) The U.S. Navy M&S MS3G. Comprised of government 
personnel (and their designates), as well as representatives from Federally Funded 
Research Centers; each NMSO-approved organization provides an empowered voting 
member to the Steering Group. [6] 

(5) The U.S. Navy M&S Standards Subgroups. Comprised of 
M&S subject matter experts from government, industry, academia, and laboratories as 
well as individual analysts. This Subgroup is focuses on Technology and Application 
Planning and Review. [6] 

(6) Other U.S. Navy M&S Projects. 

(7) Platform and Weapons Systems Acquisition Programs. 

(8) OPNAV, DoN Secretariat, Fleets, and Systems Commands. 

(9) Advisors and liaisons from the other Services’ M&S Offices 
and other DoD M&S related agencies. 
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7. U.S. Navy OT for Surface Ships 

The SURFACE Warfare Division of the Operational Test and Evaluation Force 
(OPTEVFOR) is responsible for the planning and execution of operational test and 
evaluation of U.S. Navy SURFACE ships and associated engineering, auxiliary, and 
combat systems. [7] This section discusses the M&S approaches of the LPD-17, DDG- 
51, and DD-21 (DD-X) SURFACE ships warfare programs. The VV&A processes of 
these three programs, as well as the COMOPTEVFORINST 5000.1 A process, are 
discussed in Chapter III. 

a. LPD-17 

The SAN ANTONIO Class (LPD-17) Amphibious Transport Docks is the 
product of many excellent ideas furnished by hundreds of U.S. Navy and U.S. Marine 
Corps fleet operators, maintainers, and trainers through the design for ownership process. 
[8] The 12 ships of the LPD-17 Class will replace four retiring amphibious ship classes. 
[ 8 ] 

These are versatile ships that perform the mission of amphibious 
transports, amphibious cargo ships and older dock landing ships, by incorporating both a 
flight deck and a well deck that can be ballasted and de-ballasted to support a landing 
craft. [9] The LPD-17 ships have increased vehicle and cargo carrying capacity that 
make them a key element of the 21 st century Expeditionary Strike Groups (ESG). [9] 
These ships integrate the latest in shipbuilding and warfighting technologies to support 
current and future Marine Corps aircraft, Expeditionary Fighting Vehicles (EFVs), and 
Landing craft air cushion (LCAC) or conventional landing craft. [9] 

The LPD-17 program acquisition strategy has relied heavily on the use of 
technology as a means to define requirements, explore and evaluate design alternatives, 
test and evaluate the design, streamline production sequencing, mitigate risk, and 
improving program cost and schedule control. [10] The design review process involved 
assessment of the ship design using two and three-dimensional graphic visualizations, 
generated from the LPD-17 product model. [10] The visualizations display a design 


22 



zone’s structure and systems, and are used to study space, shape, and arrangement. [10] 
The graphic visualizations also evaluate the installation sequence, interference, lines-of- 
sight, and accessibility for the purpose of design, assessment, engineering analysis, 
planning, and instruction. [10] 

b. DDG-51 

The ARLEIGH BURKE Class (DDG-51) of multi-mission, battle force 
capable guided missile destroyers provides precision engagement of targets ashore, full¬ 
dimensional protection of joint and allied forces, and dominant maneuver in the open 
oceans and littorals around the globe. [11] DDG 51s are being constructed in flights to 
incorporate technological advancements during construction: Flight I (DDG-51 to 71), 
Flight II (DDG-72 to 78) and Flight IIA (DDG-79 to 107) [12] 

In May 1998, the DDG 51 Program Office (PMS 400) agreed to accredit 
all M&S used in the Flight IIA Live Fire Test & Evaluation (LFT&E). [12] The plan 
addressed LFT&E critical issues that answered the specific M&S that will be used, and 
how that M&S will be used to address the LFT&E critical issues. [12] M&S will be used 
to model the ship, identify characteristics of weapons likely to be encountered in combat, 
predict the likelihood of being hit, the effects of hits and close detonations, and likely 
locations. [12] 

LFT&E requirements for the DDG-51 program will be in accordance with 
U.S. Code, Title 10, Subtitle A, Part IV, chapter 139, Section 2366; “Survivability and 
Lethality Testing”. [12] LFT&E consists of “realistic survivability testing” to detennine 
the system vulnerability against threats likely to be encountered in combat (conventional 
threats only). LFT&E must be done early enough in the program to allow any design 
deficiencies to be corrected before proceeding beyond low-rate initial production (or 
equivalent program milestone). [12] 
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c. DD-X (formerly DD-21) 

In November 2001, DoD announced that the DD-21 program for the future 
surface combatant had been revised and would now be known as DD-X. [13] The 
program focus would now be on a family of advanced technology SURFACE 
combatants, rather than a single ship class. [13] 

DD-X replaces the DD-21 ZUMWALT program, which was for a class of 
32 multi-mission destroyers to replace OLIVER HAZARD PERRY class frigates (FFG- 
7) and SPRUANCE class destroyers (DD-963) from 2012. [13] Unlike previous classes 
of destroyers, which were primarily to counter deep water threats, the DD-21’s primary 
mission would be to provide land attack support for ground forces and also to carry out 
traditional destroyer missions of anti-air, anti-surface and undersea warfare. [13] 

The M&S vision for the DD-21 program was to employ M&S throughout 
the DD-21 development process to provide insight during system engineering activities, 
support trade studies and risk reduction activities, and support verification of “as 
designed” system perfonnance. [14] M&S will also be utilized throughout the system 
life cycle. [14] The Smart Product Model (SPM) would be the key to achieving this 
vision, where the weapon system would be conceived, designed, built, tested, training 
and operation would be provided in a computer before cutting metal and through the life 
cycle. [14] 

The SPM is the digital version of the ship that is comprised of the product 
model data and perfonnance and behavior characteristics. [14] The SPM will be 
developed in three phases: 

(1) Phase I. Develop a Capability Specification. [14] 

(2) Phase II. Develop and deliver a system prototype. [14] 

(3) Phase III. Tailor SPM for use in manufacturing and life cycle support. 

[14] 
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D. CONCLUSION 

This chapter discussed DoD and U.S. Navy M&S strategies and provided an 
overview on the administration of M&S in the U.S. Navy. It also addressed how M&S is 
being used for three surface ship programs. The next chapter provides background on the 
VV&A processes prescribed by DoD and the U.S. Navy and discusses the VV&A 
processes of COMOPTEYFORINST 5000.1 A and three surface ship program offices. 
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III. BACKGROUND ON VERIFICATION, VALIDATION, AND 

ACCREDITATION 


A. INTRODUCTION 

This chapter provides background on the Verification, Validation, and 
Accreditation (VV&A) processes prescribed by DoD and the U.S Navy. A review is 
made of DoD’s principal VV&A instruction and of their recommended practices guide. 
Regarding the U.S. Navy, the VV&A processes of COMOPTEVFOR and three surface 
ship program offices are described. 


B. DOD VV&A PROCESSES 

1. Introduction 

In 1994, as a result of the great need for credible, cost-effective, and efficient 
M&S, the DoD directed each of the Services to develop a VV&A process. [1] As 
recounted in the Hall, Conwell & Stutzman paper, “Focusing on Credibility and 
Confidence - U.S. Navy Modeling and Simulation Verification, Validation, and 
Accreditation,” the DoD’s mandated VV&A process was intended to provide the end 
user with information that would determine whether and how the results of a model or 
simulation can be applied to decisions that would ultimately affect the warfighter. [1] 

In April 1996, DoD Instruction (DoDI) 5000.61 on “DoD Modeling and 
Simulation (M&S) Verification, Validation, and Accreditation (VV&A)” was 
established. This instruction implemented policy, assigned responsibilities and 
prescribed procedures for the VV&A of DoD M&S. [1] In November 1996, the Defense 
Modeling and Simulation Office (DMSO) followed up with the publication of the DoD 
Verification, Validation, and Accreditation Recommended Practices Guide (RPG) to 
provide instructions on how, when and under what circumstances formal VV&A 
procedures should be used. [1] 
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2. DoD Applicability and Policy 

As put forth in the instruction, DoDI 5000.61 is applicable to members of the 
Office of the Secretary of Defense (OSD), the Military Departments, the Chainnan of the 
Joint Chiefs of Staff (CJCS), the Combatant Commands, DoD’s Office of the Inspector 
General, the Defense Agencies, the DoD Field Activities, and any other organizational 
entities in DoD (that are considered to be “DoD components”). [15] All M&S that is used 
to support DoD activities are under this instruction, including those that support 
Operational Test and Evaluation (OT&E), which are subject to guidance from the OT&E 
director. [15] 

DoDI 5000.61 prescribes policy that states: 

Models and simulations used to support major DoD decision¬ 
making organizations and processes ... shall be accredited for that 
specific purpose by the DoD Component M&S Application 
Sponsor. [15] 

The instruction also prescribes that: 

Each DoD Component shall be the final authority for validating 
representations of its own forces and capabilities ... and shall be 
responsive to the other DoD Components to ensure its forces and 
capabilities are appropriately represented. [15] 

3. DoD Responsibilities 

DoDI 5000.61 outlines the responsibilities for four key positions within DoD, 
which include: 

a. the Under Secretary of Defense for Acquisition, Technology, and 
Logistics, who is required to coordinate with the DoD components, develop policies, 
plans, procedures, and DoD issuances for effective implementation and management of 
VV&A of DoD M&S; [15] 

b. the Assistant Secretary of Defense for Command, Control, 
Communications, and Intelligence, who is required to resolve validation issues and 
ensure intelligence processes and capabilities are represented appropriately; [15] 
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c. the Principal Staff Assistants (PSAs) and the Heads of the DoD 
components, who are required to plan and provide resources, as needed, to carry out the 
functional VV&A responsibilities in accordance with the component priorities; [15] and 

d. the Chairman of the Joint Chiefs of Staff, who establishes the 
VV&A policies, procedures, and guidelines to satisfy the needs of joint activities that 
reports to him/her, and who coordinates with other DoD components to establish 
procedures for the validation and accreditation of joint M&S used for joint applications. 
[15] 


4. DoD Procedures 

DoDI further states that DoD procedures require that V&V shall be incorporated 
into the development and life-cycle management process for all M&S that are currently 
being used by DoD. [15] The procedures also require that, in performing V&V for any 
federation of models, the elements must be physically connected to exchange data and 
should provide adequate, accurate, and consistent simulated representations of the models 
that addresses the mission objectives. [15] In addition, V&V information and data should 
be documented, readily available, and accessible to DoD users. Therefore, if resources 
allow, DoD components must establish an M&S VV&A repository that identifies existing 
M&S VV&A documentation and procedures that would allow DoD users to identify and 
access M&S VV&A documentation, information, and data. [15] 

5. Recommended Practices Guide (RPG) 

DoD and military services have recognized the growing significance of modeling 
and simulation for many aspects of their operations, and have prepared directives and 
guidelines to provide general instructions on how, when, and under what circumstances 
formal VV&A procedures should be employed. [16] The Recommended Practices Guide 
(RPG) is intended to facilitate the application of those directives and guidelines, and to 
promote the effective application of VV&A. [16] 
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The RPG describes the interrelated processes that make up VV&A from a number 
of perspectives, as well as describes the different roles and responsibilities of the various 
participants, the special topics associated with VV&A, and identify tools and techniques 
that are used in VV&A. [16] The RPG answers two key questions: 


a. Why is VV&A Performed? 

The RPG argues that, in order to determine whether a model or simulation 
or federation of models should be used in a given situation, its credibility (Should it be 
trusted?) must be established by evaluating its fitness for the intended use. [Ref 198] 
Verification (i.e., Did I build the thing right?), validation (i.e., Did I build the right 
thing?), and accreditation (i.e., Should it be used?) are three interrelated but distinct 
processes that collect and evaluate evidence to determine the simulation’s capabilities, 
limitations, and perfonnance relative to the real-world systems it would simulate. [16] 
The decision as to whether or not to use M&S will depend on the model’s capabilities 
and correctness, the accuracy of its results, and its usability within the specified 
application. [16] 

VV&A processes are perfonned to establish the credibility of the M&S; 
credibility depends on the model’s capability and accuracy necessary for its intended use. 
Thus, the decision on whether or not the M&S provides the necessary degree of accuracy 
depends upon the inherent characteristics of the M&S, how it will be used, and the 
significance of the decisions that may be reached from the M&S outputs. [16] 

As there are several factors that need to be considered before determining 
whether a simulation might be used, it is clear that no simple “yes/no” decision applies in 
all circumstances. [16] Therefore, though an M&S is judged suitable for one purpose in 
an organization, it does not automatically guarantee that it would be suitable for the same 
type of use in another organization. Similarly, it is also possible that the same M&S 
might not be suitable for another type of use within that same organization. [16] 
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b. Why is VV&A Important? 

The RPG also argues that the importance of VV&A is derived from the 
intended use of the M&S to which it will be applied. [16] Therefore, if an M&S is to be 
used to support training requirements, then the importance of VV&A would depend on 
the importance of the activity for which the training is being conducted, the degree of 
accuracy required for the training to be effective, and the expected degree of difficulty for 
the M&S developer in achieving that accuracy. [16] 

Continued reliance on M&S within the acquisition cycle process increases 
the financial and safety risks from erroneous or inaccurate M&S results. [16] The 
availability of definitive V&V records would help technical managers decide whether or 
not to use M&S, or to modify and re-use an existing M&S, rather than undertake 
development of a new simulation. [16] A well-defined V&V process increases the 
potential for cost savings from M&S re-use. [16] DoD is encouraged by these “factors” 
and is increasing its emphasis on employing VV&A. [16] 

Special VV&A considerations apply when an M&S may be used as a 
substitute for some prototype field testing or live fire testing, because the extent of the 
VV&A will depend on the significance of the live test that would be replaced by a 
simulated test. [16] There have been cases throughout history that illustrate what can go 
wrong when inadequate testing is performed. [16] This is where M&S can help, by 
identifying the essential areas for testing, and by prioritizing the required testing 
resources. [16] 

Finally, the RPG allows that V&V must be tailored to match the nature of 
the problem, which includes not only the situation(s) being simulated, but also the types 
of decisions that are driving the use of the M&S. [16] Every situation will be somewhat 
different from the ones that went before it - even if the type of M&S is the same, so no 
“cookbook” VV&A process can fit all situations all the time. Therefore, tailoring the 
V&V effort is a key part of the V&V process in itself. [16] 

As mentioned earlier, VV&A is the combination of three distinct 
processes, which are Verification, Validation and Accreditation. Per DoD Instruction 
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(DoDI) 5000.61 (as recounted by the RPG), the definitions for these processes are as 
follows: 


a. Verification 

The process of detennining that a model implementation and its 
associated data accurately represent the developer’s conceptual 
description and specifications. In other words, Was the thing built 
right? [16] 

b. Validation 

The process of detennining the degree to which a model and its 
associated data provide an accurate representation of the real world 
from the perspective of the intended uses of the model. This could 
also be stated as, Was the right thing built? [16] 

c. Accreditation 

The official certification that a model, simulation, or federation of 
models and simulations and its associated data is acceptable for use 
for a specific purpose. Simply put, Should it be used? [16] 


The RPG reminds VV&A stakeholders that the accreditation process is the 
official certification that an M&S and its associated data are fit for use in a specific 
application. [16] While accreditation is perceived as occurring at the end of the 
development process, the RPG argues that, in actuality, this assessment process should 
begin as early as possible so V&V and testing activities can provide the appropriate and 
sufficient information to support the accreditation decision, which is essentially the user’s 
belief in the credibility of the M&S. [16] As outlined in the RPG, the accreditation 
process is comprised of the following activities: 


a. Developing an Accreditation Plan 

Develop an accreditation plan to identify all the information needed to 
perform the accreditation assessment, schedules, resources, etc., to be used in the 
accreditation assessment. [16] 
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b. Collecting and Evaluating the Accreditation Information 

Collect and evaluate the accreditation information from the V&V efforts 
and other sources for assessment, to determine its completeness. [16] 


c. Performing an Accreditation Assessment 

Perform an accreditation assessment to determine the fitness of the 
simulation, through the evidence collected from the V&V effort and other sources, and 
prepare an accreditation report and recommendations for the user. [16] 

Once the V&V effort and the accreditation assessment are completed, the 
data collected will help identify the risks associated with using the simulation. The user 
then weighs the risks against the evidence of the simulation capabilities. [16] When 
weighing the risks, the RPG holds that five different options can be considered: 


a. Option 1 

Full accreditation, or using the simulation “as is,” which means accepting 

the risks. [16] 


b. Option 2 

Limited accreditation, or constraining the application to minimize the 

risks. [16] 


c. Option 3 

Modifying the simulation as needed to make corrections that would reduce 
the risk. This effort would increase costs and cause delays. [16] 

d. Option 4 

Requiring additional infonnation, as needed, to better understand the risks 
involved and to build confidence in the simulation’s fitness before making a decision. 
[16] 
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e. Option 5 

No accreditation decision, because it was detennined that both the risks 
and costs involved in using the simulation are too great. When no accreditation is 
determined, the user must select a different method to solve the problem. [16] 

The RPG notes that proper execution of a VV&A process involves participants in 
a number of different roles, who are identified in DoDI 5000.61. These “key players” 
and their responsibilities are defined as follows: 


a. User 

Represent the organization, group, or person responsible for the 
overall application. The user needs to solve a problem or make a 
decision and wants to use simulation to do so. The user defines the 
requirements, establishes the criteria by which simulation fitness 
will be assessed, determines what method or methods to use, 
makes the accreditation decision, and ultimately accepts the 
results. [16] 


b. Modeling and Simulation Program Manager (M&S PM) 
M&S PM is the term used to define the role responsible for 
planning and managing resources for simulation development or 
modification, directing the overall simulation effort, and 
overseeing configuration management and maintenance of the 
simulation. [16] 


c. Developer 

The role responsible for actually constructing or modifying the 
simulation, preparing the data for use in the simulation, and 
providing technical expertise regarding simulation capabilities as 
needed by the other roles. [16] 


d. Verification and Validation Agent (V& VAgent) 

The V&V Agent is responsible for providing evidence of the 
simulation’s fitness for the intended use by ensuring that all V&V 
tasks are properly carried out. [16] 


e. Accreditation Agent 
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Provides guidance to the V&V Agent to ensure that all the 
necessary evidence regarding simulation fitness for use is obtained; 
collects and assesses the evidence; and provides the results to the 
User. The Accreditation Authority is the role with the 
responsibility of making the accreditation decision. [16] 


The RPG concludes that, in order for VV&A to be successful, all the 
above players must establish and maintain a healthy working relationship. [16] 


C. COMOPTEVFORINST 5000.1A 

The Commander, Operational Test and Evaluation Force (COMOPTEVFOR) 
issued his latest policy on VV&A of M&S used for OT through COMOPTEVFOR 
Instruction (COMOPTEVFORINST) 5000.1 A of September 9, 2004. 

1. Purpose 

The purpose of COMOPTEVFORINST 5000.1 A is “to provide policy and 
process for COMOPTEVFOR to increase the potential benefit of M&S” in “the early 
planning, organizing, and execution of credible M&S programs” in support of 
Operational Testing (OT), in accordance with SECNAVINST 5200.40 & DoDI 5000.61. 
[17] 

2. Applicability and Approach 

This policy is applicable to all test assets, test planning aids, and 
post-test analysis tools that substitute for the varying 
characteristics of the system under test, supporting forces, the 
threat, other forces, or the battlespace environment for an 
operational test event or any event that is used to support 
operational test and evaluation. [17] 

The COMOPTEVFOR instruction goes on to state that, since OT is not conducted 
in real environments against real threats operated by hostile forces, it is therefore 
important that the appropriate facets of the real-world test assets (being replaced by 
M&S) are represented well enough to “generate the combat interactions that are being 
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monitored for the evaluation of combat effectiveness” of the system under test (SUT). 
[17] 

For M&S to play a role in OT, it must be credible with the tester 
and stand up to oversight. Because virtual test assets are not as 
easy for testers to intuitively assess for appropriateness as physical 
test assets, it is imperative that a well documented, understandable 
procedure is in place to characterize the use of modeled or 
simulated data in OT. [17] 


COMOPTEVFORINST 5000.1 A employs a well-defined approach for M&S 

development that facilitates the accreditation process and is in keeping with policies that 

are advocated by OSD and SECNAV. [17] 

Accreditation is supported by the specification of the verity and 
validity required for the applications to serve as test assets, and the 
planning necessary to build, configuration manage, characterize, 
and assess the utility of the applications. [17] 


3. Responsibilities 

Included in the stated objectives of COMOPTEVFORINST 5000.1 A is the need 
to cover all M&S used to supplement OT or to produce DT data that is used as entry to 
OT criteria. Consequently, there are a number of responsibilities assigned to individuals 
to perform and support those objectives and their tasks. [17] The individuals identified by 
the instruction as needed to support the various tasks are as follows: 


a. Commander, Operational Test and Evaluation Force 

(1) Provide policy and guidance for the intelligent application of 
M&S in OT. (2) Accredit the use of M&S outputs to resolve 
critical operational issues (COI) during operational evaluation 
(OPEVAL) or follow-on operational test and evaluation (FOT&E). 
[17] 


b. Assistant Chiefs of Staff, Warfare Divisions (ACOS(W)) 

(1) Provide an assessment to COMOPTEVFOR of impacts to the 
test program of using M&S data to resolve COI during OPEVAL 
or FOT&E. (2) Accredit the use of M&S data to assess COI used 
in tests other than OPEVAL and FOT&E. [17] 
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c. Operational Test Coordinators (OTC) 

(1) Assess whether appropriate program resources have been 
allocated and applied to M&S application development, 
management, and use to mitigate the risk of using M&S data in the 
OT program. (2) Make recommendations to the ACOS(W) on the 
program viability of M&S applications for use in OT. [17] 


d. Operational Test Directors (OTD) 

(1) Manage the use of M&S as a test asset for the conduct of OT. 

(2) Make recommendations to the ACOS(W) on the appropriate¬ 
ness of M&S applications to support the resolution and assessment 
ofCOIinOT. [17] 


e. Modeling and Simulation Manager (MSM) 

(1) Manage the use of M&S applications to support OT across the 
command. (2) Provide direct support to the ACOS(W), OTC, and 
OTD in executing the policy, responsibilities, processes relating to 
the use of M&S in OT. (3) Review M&S program data requested 
from PM for rigor and relevance to the application’s intended use 
within OPTEVFOR. (4) Make recommendations to the ACOS(W) 
on the technical soundness of M&S applications with respect to 
programmatics and appropriateness for use to support OT. (5) 
Maintain the corporate knowledge on the use of M&S to support 
OT and apply that knowledge to improving the policies and 
processes at OPTEVFOR. (6) Work with the command’s Staff 
Editor to maintain and update formats for M&S process 
documentation in the command’s OT&E reference library 
directory, OT&E formats folder and to update and maintain 
detailed instructions for completing the fonnats in the Modeling 
and Simulation folder. [17] 

4. Accreditation 

The policy section of COMOPTEVFORINST 5000.1 A discusses the accreditation 
approval process. Although the officers and staff of COMOPTEVFOR do not lead M&S 
efforts, they participate by generating OT acceptability requirements and providing 
operational insight at both technical and management reviews. [17] The command 
establishes and assesses the accreditation criteria for all M&S that will be used in OT, 
and often assists acquisition program offices in the development of other accreditation 
criteria for M&S. [17] COMOPTEVFOR requests that M&S accreditation decisions be 
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scheduled well ahead of the test event so as to allow alternative approaches for testing in 
the event accreditation is not granted. Consequently, PMs are advised to submit 
sufficient verification and validation (V&V) data in timely increments to support the 
accreditation. [17] The instruction states that PMs and COMOPTEVFOR officials should 
jointly detennine if assets and resources are appropriate to support testing. [17] This 
decision should be made early within the acquisition process, along with identifying the 
proper M&S requirements, which include programmatic, operational and performance 
factors. [17] 

The instruction states that M&S accreditation letters are issued by the 
accreditation authority (i.e., COMOPTEVFOR) and may include an accreditation 
assessment report that confirms that the M&S is adequate and credible for the intended 
use. [17] The level of demonstration that is acceptable for the accreditation criteria is 
determined by the MSM and status is then provided to the PM. [17] COMOPTEVFOR 
establishes the minimum fidelity requirements for the V&V data that supports 
accreditation for OT, during the development of the V&V plan, including the acceptance 
of the V&V techniques to be used, the application method, and the degree of observation 
required to build confidence in the V&V results to be assessed for accreditation. [17] 


5. Credibility and Resourcing 

COMOPTEVFORINST 5000.1 A notes that M&S development resources should 
reflect the technical expertise needed to develop, manage, and operate the VV&A 
process, as well as identify the financial and material requirements for accreditation. [17] 
These resources that are needed to support DT and OT uses must be identified as early as 
possible to enable the program to budget appropriately. [17] 

Finally, the instruction points out that the OTC will ensure that adequate 
responsibilities are identified in the test and evaluation strategies (TES) and adequate 
resources are allocated in the Test and Evaluation Master Plan (TEMP) to ensure that 
M&S applications produce credible data for use in OT (and DT events that support OT). 
[17] 
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D. VV&A PROCESSES OF U.S. NAVY SURFACE SHIPS 

This section will address the VV&A processes of the LPD-17, DDG-51, and DD- 
X (DD-21) U.S. Navy surface ships warfare programs, which were previously described 
in Chapter II. 

1. LPD-17 VV&A Process 

The LPD-17 M&S program is based upon the requirements and fundamentals 
described in the DoD/DoN 5000 series of directives and instructions and in the DMSO- 
produced VV&A RPG. [10] 

The LPD-17 VV&A Guide is tailored from the DMSO VV&A RPG and is used 
by the LPD-17 team members to define and create the VV&A activities and 
documentation required for each M&S activity. [10] The VV&A Guide assists team 
members as they establish the level of credibility for their specific M&S effort by 
providing both a process framework and a set of minimum requirements for each M&S 
activity family. [10] These requirements take a “reasonable” approach to establishing an 
M&S application’s credibility, with the ultimate goal of obtaining well defined and 
understood M&S requirements. This also includes the goals that: 

a. the capabilities, limitations, assumptions, and approximations of 
the application are documented and understood; 

b. the performance of the selected application meets the M&S 
requirements of the task; and 

c. the input data used is correct and sufficiently accurate. [10] 

2. DDG-51 W&A Process 

The PMS 400 Flight IIA LFT&E accreditation approach requires the technical 
review panel (TRP) gather data using an accreditation questionnaire and conduct reviews 
with M&S proponents that discuss model descriptions; measures of effectiveness and 
performance; variables calculated; objects and functions modeled and their limitations; 
key sources of input data; configuration management; documentation status, V&V 
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techniques used and specific V&V that were conducted; and capabilities, limitations, and 
limiting assumptions. [12] PMS 400’s Simulation Control Panel (SCP) conducts the 
analysis of capabilities, limitations, and limiting assumptions that apply to resolving the 
LFT&E issues. Their Accreditation Review Panel (ARP) assesses the risks associated 
with M&S weaknesses and makes an accreditation recommendation to the Accreditation 
Authority (AA), who is PMS 400 in for the M&S used for OT. [12] 

Some of the lessons learned from the Flight IIA Accreditation Process are: 

a. Model documentation may not be readily available for legacy 

M&S. [12] 

b. Funding was not initially provided to document the Configuration 
Management process. [12] 

c. V&V terminology and expectations needed to be better defined 
and managed. Questions, such as What is V&V? and “If M&S is accredited, does the 
process have to be repeated?” need clearer answers. [12] 

d. Not clear whether the results of previous V&V efforts were 
documented. [12] 

e. Need to better understand how the capabilities and limitations of 
the M&S will affect answering the critical issues before deciding whether or not to use a 
particular M&S. [12] 

f. Some LFT&E issues cannot be answered through M&S alone. [12] 

g. M&S is crucial to conducting a thorough evaluation of ship 
survivability. [12] 

3. DD-X (DD-21) VV&A Process 

PMS 500 has defined an accreditation process to ensure that the program office 
has acquired sufficient insight and confidence in relevant M&S to support the DD-X 
program. [18] This process is structured into four parts to illustrate how it can be tailored 
to a particular development phase. [18] 
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a. The first part is risk/resource management, where techniques are 
used to optimize resource allocation for accreditation efforts and M&S are selected and 
submitted for accreditation (after meeting PMS 500 defined criteria on prioritizing the 
M&S of interest). [18] A Simulation Control Panel (SCP) conducts an initial, 
engineering-level review to ensure that the criteria have been met, and then notifies the 
M&S proponent, ARP, and AA of the schedule for V&V review. [18] When notified, the 
M&S configuration manager begins tracking model status. During this phase, the AA 
may delegate a functional area leader as accreditation authority to determine the risk 
associated with the M&S used. [18] 

b. The second part is the insight/V&V analysis. This is when the 
SCP reviews the proponent-proposed accreditation success criteria and assesses the V&V 
information that would include V&V plans, reports, and observations associated with 
each phase of the M&S development. [18] 

c. The third part is judgment. This is when the SCP informs the ARP 
of the assessments and forwards a recommendation of accreditation that is consistent with 

the development schedule. The ARP then provides further review and solicits approval 
from the accreditation authority. [18] Accredited M&S are logged in a repository for 
possible reuse. [18] 

d. The fourth part is when the V&V review results are given as 
feedback to the M&S proponent. The feedback would include information such as 
clarifications, blunders, long-lead material, and PM approval. [18] This feedback is 
controlled by phase-dependent policies; for instance, phase II assessments of V&V 
results are retained by PMS 500 until a contractual down-selection has been made. This 
helps avoid any government influence on the competition. [18] 

4. Conclusion of Review of Surface Ship Program VV&A Processes 

The described processes indicate that some surface ship acquisition program 
managers have adopted VV&A practices that are similar to what 
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COMOPTEVFORINST.5000.1A prescribes for OT and are in line with the policies 
promulgated by OSD and SECNAV. However, from review of other processes, it 
appears that program managers, in general, still may not realize the full benefits (with 
respect to cost reduction and risk avoidance) that could be achieved by closely following 
the standards of the COMOPTEVFORINST 5000.1 A VV&A process to support their 
M&S requirements for OT. 


E. CONCLUSION 

This chapter described DoD’s VV&A processes, as well as the U.S. Navy’s 
COMOPTEVFORINST 5000.1 A VV&A process and the VV&A processes of three 
surface ship programs. The next chapter will describe the research methodology used to 
collect data for this thesis. 
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IV. RESEARCH METHODOLOGY 


A. INTRODUCTION 

This chapter describes the research methodology that was used to obtain the data 
for this thesis. The first step taken was the identification, by category, of the major 
stakeholders who participate in the VV&A of M&S used for the OT of newly acquired or 
updated U.S. Navy surface ships. Then, information and data on the practices and 
expectations of VV&A were obtained through literature reviews, forum attendances, and 
semi-structured interviews of available stakeholders. Data from the semi-structured 
interviews was analyzed and reported for each of ten interview questions by W&A 
stakeholder category and stakeholder management level. Conclusions were developed by 
a systematic review of the results against the thesis objectives and the primary and 
subsidiary research questions. Recommendations were fonned through consideration of 
the conclusions and the researchers’ general experience and expectations for the future of 
VV&A. 

B. IDENTIFICATION OF MAJOR STAKEHOLDERS 

It became clear at the onset that a key factor for analyzing this essential sector of 
the U.S. Navy’s VV&A business was the identification and categorization of the process 
stakeholders. By definition, these stakeholders “own a stake” in the VV&A process, 
whether through direct investment of their resources and the resulting expectation of a 
return on investment, or because they are responsible for ensuring that the models and 
simulations developed are credible for their intended use. Four distinct, fundamental 
roles are played by U.S. Navy stakeholders that explain their individual practices and 
shape their various expectations for the outputs and outcomes of the VV&A process used 
for OT. 


1. Tester/Accreditor 

The first of the fundamental stakeholder categories that participate in the VV&A 
of M&S used for OT of U.S. Navy surface ships is that of the Tester/Accreditor. This is 
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a single stakeholder category that nevertheless encompasses a dual function: The 
stakeholder, in order to establish the suitability and effectiveness military items, either 
conducts physical testing of hardware in real world enviromnents or accredits models and 
simulations used in virtual environments. 3 Making the operational tester also be the 
accreditor of the M&S employed for OT is essential: By maintaining both functions at 
one activity, the command has the complete authority it needs to officially state that a 
warfighting system’s suitability and effectiveness has been demonstrated. For U.S. Navy 
OT, the Tester/Accreditor stake-holder role is fulfilled by the Operational Test Directors 
(OTDs) at COMOPTEVFOR, as they are solely responsible for the operational test and 
evaluation of all U.S. Navy warfighting systems, including surface ships, and the 
accreditation of all M&S used in support of their OT. Of further relevance, 
COMOPTEVFOR is organizationally independent of the other VV&A stakeholder 
groups within the U.S. Navy. Thus, the accrediting role played by COMOPTEVFOR 
(through the OTDs) in this VV&A process is free of those interests that constrain the 
other stakeholders. 

2. Resource Sponsor 

The second stakeholder group in this VV&A process is that of Resource Sponsor. 
(Other titles frequently used in U.S military organizations for this group are 
“Requirements Officer” and “Combat Developer.”) The Resource Sponsor identifies the 
operational requirement and/or capability desired for the warfighting system under his/ 
her cognizance and obtains the funding needed to acquire the system. Regarding the 
subject at hand, the Resource Sponsor must see that adequate funding and time are 
budgeted for the conduct of the VV&A and agree that developing M&S products will 
best demonstrate the satisfaction of requirements and/or the fulfillment of warfighting 
capabilities. For the U.S. Navy, the staff codes responsible for overseeing particular 


3 The same person, i.e., the Operational Test Director (OTD) under COMOPTEVFOR, is specifically 
responsible to COMOPTEVFOR for ensuring that both physical tests and models/simulations address the 
COIs for the OT of a particular U.S. Navy surface ship (and/or its weapons systems). A single focal point 
ensures that the COIs are sufficiently and completely addressed. The OTD is assisted by the COMOPTEV¬ 
FOR VV&A Manager in reviewing and recommending M&S used for OT to their boss, COMOPTEVFOR. 
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warfighting platforms within the Chief of Naval Operations (OPNAV) play this 
stakeholder role. In the case of surface ships, Code N76 is the Resource Sponsor. 
Because of his/her oversight function in the acquisition of surface ships, the N76 
Resource Sponsor may often play a mediating role with respect to the other W&A 
stakeholders. 


3. Program Manager 

The third stakeholder group in this VV&A process is that of Program Manager 
(PM). The PM is the senior manager in the acquisition office that charters the 
development of the models and simulations and proposes that they be used (instead of 
testing with physical assets) to demonstrate satisfaction of OT requirements. Thus, the 
PM is also known as the M&S proponent in the VV&A for OT process. The PM, 
through his/her program office, is responsible for delivering a warfighting system within 
budget and schedule that meets prescribed requirements, and is thereby granted a certain 
amount of discretion within these constraints to accomplish the program’s mission. 
Therefore, he/she may consider a number of options, including whether to propose 
physical or virtual testing for the fulfillment of OT. At the same time, he/she must 
ultimately convince the U.S. Navy surface ship acceptance authorities to accept the 
program’s deliverables, whether they be demonstrated by physical or virtual testing. 
Thus, the PM acts as the prime mover in the VV&A process by proposing action to the 
other stakeholders. 

4. VV&A Standards Official 

The fourth player in the VV&A of M&S used for OT of U.S. Navy surface ships 
is that of the “VV&A Standards Official” category. This group is interested in seeing 
that all of the VV&A stakeholders are following a standard process that is documented 
and repeatable. The Navy Modeling and Simulation Office (NMSO - fonnerly 
NAVMSMO - Navy Modeling and Simulation Management Office) performs this 
function within the U.S. Navy. Though a part of the U.S. Navy’s OPNAV organization, 
this group operates independently of the Resource Sponsors (as well as the other 
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stakeholder groups), but supports all of the other stakeholders in communicating and 
encouraging the use of standards in VV&A. Figure 1 depicts the basic relationships 
between the four stakeholder groups and shows (as depicted by the dotted line) that the 
VV&A Standards Official group informs the other three. 



Figure 1. Relationships between OT VV&A Stakeholder Groups within U.S. Navy 

C. LITERATURE REVIEWS 

Research began by identifying the published sources of information and data on 
the practices and expectations of VV&A for M&S used to satisfy OT requirements of 
U.S. Navy surface ships (and ship weapons) that are completing their acquisition process. 
Published information was located on the official websites of the U.S. Navy and 
Department of Defense (DoD) stakeholders and downloaded. On several occasions, it 
was more expedient to directly request that the stakeholders provide the documents as e- 
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mail attachments or hardcopies delivered in person at forums. Since COMOPTEVFOR 
accredits the models and simulations used to satisfy the requirements of operational test 
and evaluation, it was paramount to begin the relevant literature reviews at this 
command. Paragraphs 1 and 2, below, identify the key documents. OPNAV, the Navy 
Secretariat (SECNAV), and DoD publish documents that provide overarching VV&A 
policy and context for the policy issued by COMOPTEVFOR. Paragraph 3, below, 
discusses this area of the literature review. Next, a review was conducted of the VV&A 
and M&S process plans released by U.S. Navy surface ship acquisition programs. These 
process plans were either published on the programs’ websites or released to the 
researchers by the program offices. Finally, the researchers obtained important insights 
on the VV&A process through review of related professional publications, many of 
which were authored by the stakeholders that participate in the VV&A of U.S. Navy 
surface ship M&S. Paragraph 5, below, covers this review. In all, five areas of literature 
review were conducted to obtain the relevant information. 

1. Review of COMOPTEVFORINST 5000.1A, dated SEP 9 2004 

A review was conducted of the recently updated COMOPTEVFOR Instruction 
5000.1 A of 9 September 2004, with the subject, “Use of Modeling and Simulation 
(M&S) in Operational Test (OT).” [17] Chapter III of this thesis provides a detailed 
description of this instruction. The Instruction cancels the previous instruction (dated 5 
September 1995), which is described below. 

2. Review of COMOPTEVFORINST 5000.1, dated SEP 5 1995 

To appreciate what was changed by the update, a review was conducted of the 
superseded instruction, COMOPTEVFOR Instruction 5000.1 of 5 September 1995, and 
its supplementary instructions, COMOPTEVFOR Memorandum 3980 Ser 00T/312 of 29 
April 1998 and COMOPTEVFOR Policy and Information Notice 99-01 of 13 May 1999. 
Most of the experience of the stakeholders interviewed was in compliance with this 
instruction and its two supplements. While the title of this earlier instruction is 
essentially identical to its replacement, the stated purpose of COMOPTEVFOR 
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Instruction 5000.1 of 5 September 1995 is more lengthy and detailed. This instruction 
was issued at the time to provide interim policy and guidance, with the expectation that it 
would be modified when “... subordinate instructions are published ....” [19] 
(Furthermore, one of the three references cited was still in draft fonn.) The instruction 
included six enclosures (which made the whole document 28 pages long) that provided a 
glossary of terms and a range of sample documents for an acceptance criteria matrix, a 
model management plan, a memorandum of agreement for Verification and Validation 
(V&V) of M&S between independent agencies, some organizational schematics for 
Simulation Management Boards (SMB) and Simulation Control Panels (SCP), and an 
end-to-end accreditation process flowchart. Regarding COMOPTEVFOR’s two 
supplemental documents: the 29 April 1998 memorandum discussed the “re¬ 
accreditation” [20] of previously accredited M&S and clarified the Command’s role in 
the accreditation of M&S that are components of systems under test; and the 13 May 
1999 Policy and Information Notice provided “guidance in accurately documenting the 
accreditation processes and results ...” [21] including three enclosures for (1) 
accreditation plan outline, (2) accreditation letter outline, and (3) accreditation 
assessment report. 


3. Review of Related U.S, Department of Defense Policy Documents 

Reviews were conducted of other VV&A, M&S, and OT documents promulgated 
by COMOPTEVFOR, as well as the related directives, instructions, guidance, and 
information notes issued by OPNAV, SECNAV, DOT&E, and OUSD. These documents 
provide overarching policy, guidance, and information on VV&A, M&S, OT, and the 
acquisition of major defense systems. As such, they provide the “support net” that 
enables compliance with COMOPTEVFORINST 5000.1 A of 9 September 2004. 
Chapter III provides detailed descriptions of two of the OUSD documents, DoD 
Instruction (DoDI) 5000.61 on “DoD Modeling and Simulation (M&S) Verification, 
Validation, and Accreditation (VV&A)” of 13 May 2003 and the VV&A “Recommended 
Practices Guide” of 4 August 2004 
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4. Review of U.S. Navy Surface Ship Acquisition Program Publications 

Reviews were conducted of the VV&A and M&S process plans released by U.S. 
Navy surface ship acquisition programs. Most of the program offices contacted by the 
researchers had developed a program specific VV&A process chart that was used to 
guide program participants through their VV&A program and inform their Resource 
Sponsors and COMOPTEVFOR. These process plans were either published on the 
programs’ websites or released to the researchers by the program offices. Chapter III 
provides brief descriptions of three of the VV&A efforts of U.S. Navy surface ship 
acquisition programs. 

5. Review of Related Professional Publications 

Reviews were conducted of related professional websites, forum briefings, 
symposium papers, journal articles, books, theses, and studies. Those publications that 
were found to provide additional insight to the mission of VV&A for the U.S. Navy are 
listed in the Bibliography. Chapter III cited a number of these publications in describing 
DoD and U.S. Navy VV&A processes. 


D. VV&A FORUM ATTENDANCE 

The research team attended five VV&A Forums (formerly: TWGs - Technical 
Working Groups) co-hosted by the Navy Modeling and Simulation Office (NMSO) and 
the U.S. Navy (or other U.S. defense command) where the forum was held. The thesis 
advisor attended the first of the forums and introduced the research team to the forum. 
The research students attended four subsequent forums. The forums attended by the 
research team were: 

1. VV&A Forum #16, on 3 March 2004, in San Diego. CA 

The NMSO (formerly NAVMSMO) VV&A Technical Working Group (TWG) 
held a workshop (now called a “VV&A Forum”) on 3 March 2004 at the SPAWAR 
Systems Center in San Diego, CA. The purpose of the forum was to allow the U.S. Navy 
M&S communities an opportunity to stay current on the latest VV&A policy, guidance, 
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resources, and tools available. In addition to briefings on current VV&A topics, an open 

discussion of the VV&A communities’ needs and issues was facilitated, in order to 

identify commonalities and possible solutions. [22] The thesis advisor attended and gave 

a briefing entitled, “Studying the A in VV&A.” The thesis research team was introduced 

and the research motivation, background, research question, methodology, and future 

steps of the thesis research were briefed. As the minutes report, the presentation 

prompted forum discussion, raising topics such as 

... the need to look at the larger issues of M&S re-use given that 
the accreditation authority is far removed from the original M&S 
development process ... [and] the need to account for funding and 
resources regardless of the status of the M&S development status. 

... In the same vein, questions were posed on how to ensure the 
accreditation of new models given the difficulty of accrediting 
legacy models. The presenter pointed out the benefits of peer 
review.” [22] 

2. VV&A Forum #17, on 7 May 2004, in West Bethesda, MD. 

The thesis researchers attended the NMSO VV&A Forum on 7 May 2004 at the 
Carderock Division of the NAVSEA Warfare Center in West Bethesda, MD. VV&A 
Forum #17 provided the U.S. Navy M&S communities an opportunity to keep abreast of 
the latest VV&A policy, guidance, resources, and tools available. In addition to briefings 
on current VV&A topics, an open discussion of the VV&A communities’ needs and 
issues was facilitated, in order to identify commonalities and possible solutions. 
COMOPTEVFOR’s VV&A Manager presented a briefing on the philosophy of OT and 
the use of M&S to represent aspects of the systems under test, threat forces, and the 
operating environment. The presentation also indicated COMOPTEVOR’s plans to 
update COMOPTEVORINST 5000.1 by releasing COMOPTEVFORINST 5000.1 A 
within the year. [23] 


3. VV&A Forum #18, on 14 September 2004, in Norfolk, VA. 

The thesis researchers attended the NMSO VV&A Forum on 14 September 2004 
at the Commander Operational Test Force (COMOPTEVFOR) Headquarters in Norfolk, 
VA. VV&A Forum #18 provided the U.S. Navy M&S communities an opportunity to 
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obtain updates on recent VV&A policy, guidance, resources, and tools. In addition to 
briefings on current VV&A topics, an open discussion of the VV&A communities' needs 
and issues was facilitated, in order to identify commonalities and possible solutions. In 
addition to presentations on current VV&A topics, a round table discussion followed and 
volunteers were solicited for reviewing the draft DoN M&S VV&A Implementation 
Handbook, Volume III (Case Studies and Lessons Learned). COMOPTEVFOR’s VV&A 
Manager announced that COMOPTEVFORINST 5000.1 A was just signed (on 9 
September 2004) and distributed hardcopies to forum participants. The thesis researchers 
presented a brief to the forum that discussed the primary research question and the 
research methodology, and requested interview participants for the thesis research. The 
issue of VV&A “practice versus expectations” was briefed and the key VV&A policy 
documents (for literature review) and the major stakeholder categories (formulated by the 
researchers) were identified. [24] 

4. VV&A Forum #21, on 11-12 May 2005, in Dahlgren, VA. 

The thesis researchers attended the NMSO VV&A Forum on 11-12 May 2005 at 
the Dahlgren Division of the NAVSEA Warfare Center in Dahlgren, VA. VV&A Forum 
#21 provided the U.S. Navy M&S communities an opportunity to leam the latest 
regarding VV&A policy, guidance, resources, and tools available and to discuss 
difficulties and areas of concern regarding program-specific VV&A efforts. This forum 
focused on DoN M&S VV&A efforts and M&S requirements, with the goal of 
facilitating greater awareness of standards, best practices, and areas of continuing 
concern. [25] The NAVSEA Dahlgren Accreditation Team (NDAT) presented their 
VV&A process, which included detailed process charts that were adapted (i.e., tailored) 
to the specific requirements of the various surface ship (and weapons) acquisition 
programs supported. 

5. VV&A Forum #23, on 28 Feb. - 1 March 2006, in Chesapeake, VA. 

The thesis researchers attended the NMSO VV&A Forum from 28 February 2006 
to 1 March 2006 at the U.S. Joint Forces Command (USJFCOM) in Chesapeake, VA. 
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VV&A Forum #23 provided the U.S. Navy, the other U.S. military services, and the 
USJFCOM M&S communities an opportunity to gain new insights on VV&A policy, 
guidance, resources, and tools available and to discuss difficulties and areas of concern 
regarding program-specific VV&A efforts. The first day of the forum concentrated on 
DoN, DoD, and Joint M&S VV&A efforts with the aim of bringing greater awareness of 
standards, best practices, and areas of continuing concern in the areas of experimentation 
and testing and evaluation. The second day was dedicated to specific “breakout” sessions 
that provided focused discussion on: (1) Experimentation for M&S and special 
considerations for VV&A; and (2) M&S VV&A in testing and evaluation. [26] 


E. SEMI-STRUCTURED INTERVIEWS 

Early in the planning phase of the thesis, the researchers developed the basic 
concept of conducting interviews of stakeholders, using a questionnaire, to develop data 
that would be useful in answering the research questions. In the semi-structured 
interview, the interviewer would pose a set of standard questions to the interviewee, but 
would also invite the interviewee to comment on any additional VV&A related items that 
seemed important. Thus, a mix of structured data and free comments would be expected 
from the interviewees. A five phase approach was employed to develop the data under 
the semi-structured interview methodology. 

1. Pilot Questionnaire 

The researchers developed a “pilot questionnaire” that was used to obtain initial 
feedback from a representative sample of participants in the VV&A of M&S used for the 
OT of newly acquired or updated U.S. Navy surface ships. This questionnaire targeted 
the PM stakeholder group and was composed of 21 questions, which ranged from its first 
question: 

“What surface ship acquisition program(s) did you work on 
or assist that used M&S tools (models) to support OT? ” 
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... that was designed to capture the VV&A experience of the stakeholder, to its last 
question: 


In your opinion, to what extent would the distinctly 
different organizational structures of U.S. Navy surface 
ship acquisition programs found at NAVSEA impact a 
standard implementation of the VV&A process advocated 
by COMOPTEVFORINST 5000.1? 

... that was fashioned to obtain a potential explanation for not following the standard 
implementation of COMOPTEVFORINST 5000.1. The researchers expected that 
stakeholder feedback on the pilot questionnaire would provide information needed to 
design the final questionnaire. 

2. Pilot Interview 

Six PM stakeholder VV&A process participants were interviewed using the pilot 
questionnaire. Four of the VV&A participants were interviewed as a group, since they 
were located at the same U.S. Navy activity and it was more convenient to schedule a 
meeting at the facility (as the researchers were required to travel to conduct the 
interviews). The participants completed the questionnaires before the interviews and 
used the meeting with the researchers to discuss or clarity their written answers. The 
researchers learned from the interviews and the data that the questionnaire/interview 
process would work more efficiently if the number of questions were reduced. It was 
recognized that 24 questions would produce too much data to process (for the intended 
number of stakeholders to be interviewed) using the pilot questionnaire. Also, it was 
realized that each question should produce answers with a numerical score as well as a 
narrative comment. The numerical score would assist in statistical analysis of the 
stakeholder’s answers and get the stakeholder to commit to the assignment of a grade 
(“1” through “5”) to the answer provided. At the same time, the researchers still desired 
narrative comments, since they would provide valuable insights as to why they assigned a 
certain score to their answer. Further, the researcher expected that many of the comments 
would provide input for the conclusions and recommendations expected from the thesis. 
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Review and reflection on the substance of the data collected by pilot questionnaire 
influenced the thesis research questions and design of the ten questions administered by 
the final questionnaire. The researchers, by interviewing just PM stakeholders in the 
pilot, realized the importance of also interviewing Tester/Accreditor and Resource 
Sponsor stakeholders (in roughly equal numbers to the PMs interviewed), and saw the 
additional benefit of interviewing at least one VV&A Standards Official stakeholder. By 
this approach, the researchers expected to collect data that would be more representative 
of the perspectives held by VV&A community at large. 


3. Final Questionnaire 

The researchers designed a final questionnaire containing ten questions, each of 
which requested the stakeholder to answer the question by assigning a score between 1 
and 5 (inclusive) and providing a comment that explained the score. (An exception to the 
1:1 score/comment answer pairing was Question 1, which asked two subsidiary questions 
in order to elicit additional comments for the score assigned.) Extra space was provided 
on the questionnaire for the stakeholders to express their opinions on the VV&A process, 
regardless of areas suggested by the questions. These three features of the questionnaire: 
quantitative scores to specific questions, narrative comments on the same questions, and 
freedom to make additional comments, provided the researchers with an instrument for 
conducting semi-structured interviews of stakeholders. The ten questions are provided in 
the Appendix. 

The basic purpose of the ten questions was to obtain data from representative, 
experienced stakeholders that answer the primary thesis question and the four subsidiary 
questions (answers which address the objectives of the thesis). Of the ten questions, the 
first was designed to capture the level of experience with the COMOPTEVFOR VV&A 
instruction and thus support the credibility of the stakeholders’ answers to the remaining 
questions. The second question solicited the stakeholders’ basic perceptions of overall 
success in complying with COMOPTEVFORINST 5000.1, which went to the heart of the 
primary thesis question. The next three questions (numbers 3 through 5) individually 
requested the stakeholders’ opinions of the benefits, costs, and risks associated with 
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instruction compliance. Conversely, questions 6 through 8 inquired about the 
stakeholders’ opinions of the benefits, costs, and risks associated with instruction non- 
compliance. Thus, this middle set of six questions were designed to cover all sides of the 
issue and not presuppose whether all stakeholders saw benefits (vs. liabilities) to 
following the COMOPTEVFOR prescribed VV&A process, as practiced. Then question 
number 9 was asked, in recognition that COMOPTEVFOR “recently” updated their 
instruction. It was anticipated that most of the stakeholders would report that the 
majority of their VV&A case experience had occurred under the old instruction. Yet the 
researchers also expected that most stakeholders would be aware of what changed in the 
new instruction and would be prepared to comment on whether the new instruction 
represented an improvement in the way VV&A is conducted. Finally, question 10 was 
designed to obtain perspectives on opportunities (if any) for further revision of the 
instruction. 

4. Semi-Structured Interviews 

Using the final questionnaire, the researchers interviewed fourteen stakeholders 
who participated in the VV&A of M&S for OT of U.S. Navy Surface Ships through one 
of these four stakeholder roles: (a) Tester/Accreditor; (b) Resource Sponsor: (c) Program 
Manager; and (d) VV&A Standards Official. Additionally, each stakeholder’s relative 
level of management responsibility within their command - as either (i) Lower, (ii) 
Middle, or (iii) Upper management - was identified. The role / level breakout of the 
stakeholders interviewed is shown in Table A. 1-0 of the Appendix. 

The researchers, for the most part, contacted the stakeholders by e-mail to arrange 
the interviews. The questionnaire was e-mailed ahead of the interviews. Many of the 
stakeholders filled out their questionnaires in advance of their interview meeting. The 
interviews were conducted on a non-attribution basis, where the stakeholders’ names and 
U.S. Navy commands were not identified in order to protect their confidentiality. The 
researchers believed that there would be a better chance that the stakeholders would 
provide frank opinions if their names or activities were not to be published. The 
researcher recorded their notes of the interview on the questionnaire for each stakeholder. 
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Thus, some of the data recorded on the questionnaire was written by the stakeholder and 
some was recorded by the researcher. After administering each question in order, the 
stakeholder was encouraged to bring up any other VV&A issue that came to mind. These 
comments were recorded in the space below question 10 (or, if more room was needed, 
on the back side of the last sheet of the questionnaire). Every attempt was made by the 
researchers to get the stakeholders to provide complete answers to the questions. 


5. Data Processing 

The researchers reviewed the questionnaires and transferred the numerical scores 
and the raw comments to an MS Excel spreadsheet to create a “Results Matrix.” Each 
stakeholder (using a letter designator) was assigned a row. Each answer was assigned a 
column. The scores for each question were tabulated down the question’s column, 
summed, and averaged for the question. The individual raw comment was loaded into 
the cell adjacent to the numerical score. This provided the analyst the facility of seeing 
the comment with the score. 

a. Identification of the Stakeholder 

To protect their privacy, each stakeholder was assigned a letter designator 
(e.g., [A] ) as their interview data was processed into the Results Matrix. Furthermore, 
the stakeholder category (e.g., Tester/Accreditor {TA} ) and management level (e.g., 
Lower Management {lm} ) was identified using the letter designators in brackets. 

b. Recognition of the Scores 

The scores recorded on the questionnaires were transferred to the Results 
Matrix. Occasionally, a stakeholder would give two scores for a given question because 
he/she had more than one point to make in their answer. This was usually explained in 
the comment they provided with the scores. The researchers reviewed the comment and 
selected the score that, in their interpretation, better answered the true sense of the 
question. 
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c. Editing of the Comments 

The raw data recorded on the questionnaire was transferred to the Results 
Matrix. After the matrix was completed and reviewed for trends, the comments were 
transferred to an MS Word document for editing. By this, the comments were corrected 
for spelling, grammar, use of undefined acronyms, use of awkward phrases, attribution to 
specific individuals or commands. Every effort was taken on the part of the analysts to 
preserve the genuine sense of the individual comments. The output of the comment 
editing process are the comments provided in the Appendix. 


F. ANALYSIS OF DATA 

For each of the ten interview questions, the following analysis was performed: 

1. Averaging of Score-Answers 

The score-answers from all the stakeholders were summed and divided by the 
number of stakeholders to produce the average score for the question. Then, average 
scores were calculated for each of the four stakeholder roles and three management 
levels. These averages were reported in a Table. Trends were detennined from this data 
that reflected the basic opinions of the stakeholders by their stakeholder roles and 
management level. These were discussed in narrative form. 

2. Grouping of Comment-Answers 

The comment-answers from the stakeholders were reviewed to determine whether 
a consensus or divergence of opinions was apparent. Majority and minority opinions 
were identified. Quotes representative of the group opinions were selected, or 
paraphrased summaries provided. Particularly exceptional comments were highlighted. 

3. Sorting of Score-Answers 

Individual stakeholder score-answers were sorted by score and counted. The 
score counts were reported in a Table. The count by score level was reviewed for overall 
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distribution of scores and discussed in narrative form. Correlations between the score 
distribution and comments were considered and discussed. 

G. DEVELOPMENT OF CONCLUSIONS 

Conclusions were reached for each question and recorded in Chapter V. These 
conclusions were developed through review of each question’s numerical trends and 
consideration of the summarized or highlighted comments. Then, in Chapter VI, overall 
conclusions were developed that addressed the thesis objectives and answers were 
provided to the primary and subsidiary research questions. These conclusions combined 
the results of the individual questions and analyzed for global trends across the data. 
Ultimately, the purpose of the literature reviews, forum attendances, and the semi- 
structured interviews was to glean and combine the data needed to address the thesis 
objectives and answer the research questions. These outputs were offered as the main 
conclusions of the thesis. 

H. DEVELOPMENT OF RECOMMENDATIONS 

Recommendations were developed through consideration of the conclusions 
discussed in Chapter VI. In addition, the researchers’ own opinions, drawing upon the 
lessons learned from the data, were indicated. The researchers pointed to opportunities 
for additional research that became apparent from the research but were not pursued due 
to the limitations of time and resources available. 

I. CONCLUSION 

This chapter described the eight part methodology used in researching the thesis. 
The first step taken was the identification of the major stakeholders who participate in 
this arena of the U.S. Navy’s VV&A process. Second, information and data on the 
practices and expectations of VV&A were obtained from literature reviews. Attendance 
at the VV&A forum constituted the third step. Fourth, the researchers developed a 
questionnaire and administered semi-structured interviews of available stakeholders. 
Fifth, the data from the semi-structured interviews was processed into a Results Matrix. 
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Sixth, the data was analyzed for each of ten interview questions by VV&A stakeholder 
category and stakeholder management level to identify trends, etc. By the seventh step, 
conclusions were developed for both the individual questions and for answering the 
primary and subsidiary research questions. Finally, individual and global 
recommendations were produced, accomplishing the eighth and final step of the research 
methodology. The next chapter will analyze the data obtained by the semi-structured 
interviews, providing the results of the thesis. 
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V. RESULTS 


A. INTRODUCTION 

This chapter reports the analysis of the data (reported in Chapter IV) that was 
developed through the administration of the questionnaire. Data from Question 1 
establishes each stakeholder’s experience with the COMOPTEVFOR instruction. 
Question 2 obtains each stakeholder’s assessment of the level of compliance observed 
with the instruction and some supporting evidence for the assessment. Questions 3, 4, 
and 5 solicit the stakeholder regarding the perceived benefits, costs, and risks associated 
with successfully complying with the instruction. Conversely, Questions 6, 7, and 8 
request each stakeholder’s opinion regarding the perceived benefits, costs, and risks 
associated with not complying with the instruction. (Questions 2 through 8 are meant to 
capture all “sides” of the compliance issue, depending on the stakeholder’s perspective. 
As such, some of these questions may not make sense to every stakeholder.) Questions 9 
and 10 elicit comments and recommendations for improvements to the COMOPTEVFOR 
VV&A instruction. 

B. DESCRIPTION 

A six part procedure was used to analyze the stakeholder data: 

1. Purpose of Question : The purpose of each question is reviewed. 

2. Scores Observed : The scores are tabulated and an overall average score is 
calculated for each question. Further, the average score for each stakeholder role and 
each management level are calculated for each question. These averages are shown in a 
table for each question. Observations are made about what these averages indicate about 
the stakeholders' answers to the question. 

3. Stakeholder Comments : The stakeholders’ comment-answers to each 
question are discussed where: (a) consensus was obtained or views diverged; (b) majority 
and minority opinions were evident; and (c) noteworthy ideas were offered. 


61 



4. Score / Comment Associations : A table is provided that shows the 
number of stakeholders that gave a particular score. Observations are made when certain 
kinds of comments correlate with particular scores. 

5. Conclusions : Closing observations are made regarding the stakeholders’ 
answers to the question. 

6 . Recommendations : Suggestions for VV&A process improvements are 
offered within the context of the question. 

The above procedure was performed for all ten questions. Overarching 
observations (trends, etc., observed across the answers to all the questions), global 
conclusions, and final recommendations are discussed in Chapter VI. 

The results of each of the ten interview questions were analyzed, as described in 
the following ten sections, C through L. 

C. ANALYSIS OF QUESTION #1 

Question #1 asked: “To what extent are you familiar with the VV&A process 
required by the newly updated COMOPTEVFORINST 5000.1 A of 9 September 2004, as 
it applies to USN surface ship M&S used for OT&E?” 

Table 1 shows the selection of scores that each stakeholder was able to choose 
from in answering the question in assigning a level of familiarity with the instruction. 


Choices of Score-Answers for Question #1 

Score: 

“5” 

“ 4 ” 

“3” 

“ 2 ” 

66 55 

Description: 

“very 

familiar” 

“moderately 

familiar” 

“somewhat 

familiar” 

“moderately 

unfamiliar” 

“very 

unfamiliar” 


Table 1. Choices of Score-Answers for Question #1 
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1. Purpose of Question #1: 

Question #1 seeks to establish each stakeholder’s experience with the 
COMOPTEVFOR VV&A instruction. Among other insights, the data obtained is 
expected to help establish the context of the stakeholder’s answers to the remaining 
questions. 

2. Scores Observed for Question #1: 

Table 2 shows the average score-answers to Question #1. The overall average 
score is 3.8, which indicates that the typical stakeholder surveyed is “moderately 
familiar” with the instruction. 


Average Score-Answers to Question #1 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

5.0 

5.0 

5.0 

5.0 

Resource Sponsor 

2.0 

1.0 

2.0 

1.7 

Program Manager 

4.0 

3.0 

5.0 

3.8 

VV&A Standards Official 



4.0 

4.0 

Avg. Score within Level 

4.0 

3.0 

4.2 

3.8 


Table 2. Stakeholder Familiarity with VV&A Instruction 


Those performing the Tester/ Accreditor role consistently answered “very familiar” and 
scored a 5.0 average, the highest score among the stakeholder role categories. On the 
other hand, those performing the Resource Sponsor role scored an average of 1.7, the 
lowest of the groups, indicating that they are typically “moderately unfamiliar” with the 
instruction. Those performing the Program Manager role scored a 3.8 average (which is 
the same as the overall average), indicating that they are typically “moderately familiar” 
with the instruction. The person representing the VV&A Standards Official role 
provided a score of 4, indicating that he/she was “moderately familiar” with the 
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instruction. The average scores within the lower, middle, and upper management levels 
are 4.0, 3.0, and 4.2 respectively. These indicate that stakeholders at the lower and upper 
management levels were “moderately familiar” with the instruction, whereas stakeholders 
at the middle management level are “somewhat familiar.” 

3. Stakeholder Comments for Question #1: 

The stakeholders participated in a wide variety of VV&A cases for OT&E of 
USN surface ships, including efforts that supported M&S of: Rolling Airframe Missile 
(RAM); Tomahawk Weapons System; Littoral Combat Ship (LCS); CVN-21; LPD-17; 
DD(X); ASCM Probability of Raid Annihilation (PRA) Test Bed; USS Virginia (SSN 
774) Transient Shock Analysis (TSA); Joint Lightweight Standoff Chemical Agent 
Detector (JLSCAD); Netcentric Warfare Knowledge Sharing; LHA-6; Threat D Suite; 
CEC; ESSM; SPYID(V); and ABMD. Most stakeholders listed participation in at least 
three of these efforts; and most of these efforts were listed by more than one of the 
stakeholders interviewed. 

The stakeholders performed particular roles in these VV&A cases, including: 
Operational Test Director (OTD); participant on Simulation Control Panels (SCPs); 
reviewer of VV&A documents; producer of the ‘Guide for U.S. Navy Surface Ship 
VV&A of M&S for OT&E’; manager of COMOPTEVFOR’s VV&A program; author / 
contributor / reviewer / approver of COMOPTEVFORINST 5000.1 A (issued 9 
September 2004); developer of long-term strategic policy with respect to the U.S. Navy 
and OT&E; resource sponsor (funding) and requirements definer (via Capabilities Design 
Documents) for U.S. Navy surface ships and/or their weapons systems; arbitrator 
between Program Manager and Tester/Accreditor with regard to balancing M&S costs 
and risks; oversight of surface ship Test & Evaluation Master Plan (TEMP) process; 
oversight of the U.S. Navy’s T&E efforts (including M&S for T&E); project engineer / 
program analyst for surface ship acquisition program office developing M&S; developer 
of M&S and contributor to VV&A documentation; manager of the VV&A process for 
U.S. Navy surface ship / weapons system acquisition Program Manager; oversight of 
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PM’s VV&A process; oversight of the U.S. Navy’s VV&A processes; and communicator 
/ coordinator of the U.S. Navy’s VV&A standards. 

The older (superseded) COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for most the cases contributed by the stakeholders. However, the newer 
instruction 5000.1 A was in effect for the cases started since 9 September 2004. 

4. Score / Comment Associations for Question #1: 

No particular correlation was seen between the stakeholder scores and the types of 
experiences (that is, the kind of ship or weapon system VV&A’ed) cited by the 
stakeholders. Table 3 shows the distribution of score-answers for Question #1. 
However, there were strong associations between scores of 5 (of being “very familiar” 
with the instruction) and the particular roles of (a) Tester/Accreditor reviewer of VV&A 
documents; (b) manager of a Command’s VV&A program; and (c) contributor / reviewer 
of COMOPTEVFORINST 5000.1 A. There were also strong associations between the 
lowest scores of 2 and 1 (of either being “moderately unfamiliar” or “very unfamiliar” 
with the instruction) and the particular roles of (a) providing program funding / defining 
acquisition requirements and (b) arbitrating between the program managers and the 
Tester/Accreditors with regard to balancing M&S costs and risks. 


Distribution of Question #1 Score-Answers 

Level of Familiarity with VV&A Instruction 

Count 

Stakeholders who provided a score of “5” 

6 

Stakeholders who provided a score of “4” 

4 

Stakeholders who provided a score of “3” 

0 

Stakeholders who provided a score of “2” 

3 

Stakeholders who provided a score of “1” 

1 

Total 

14 


Table 3. Distribution of Question #1 Scores-Answers 
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5. Conclusions for Question #1: 

The typical stakeholder surveyed is “moderately familiar” with the instruction. 
This is to be expected, given that stakeholders interviewed had key roles in the VV&A of 
M&S used for OT&E. It would be surprising if the participants in this application of 
VV&A were not familiar with the instruction since they would be expected to refer to it 
routinely in the course of performing their respective roles in the VV&A process. The 
contrary turns out to be the case only for Resource Sponsors, who typically indicated that 
they are “moderately unfamiliar” with the instruction. We learned from the interviews 
that the typical Resource Sponsor does not believe that “intimate” familiarity with the 
COMOPTEVFOR VV&A instruction is necessary to their role. Rather, they believe their 
function is to be generally familiar with the VV&A requirement so as to help balance the 
high technical expectations for the M&S, held by the Tester/Accreditors, with the desire 
to control the cost and schedule to produce the M&S, held by the Program Managers. 

D. ANALYSIS OF QUESTION #2: 

Question #2 asked: “What level of compliance with the (effective) instruction do 
you think was achieved in this case?” 

Table 4 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of compliance with the instruction. 


Choices of Score-Answers for Question #2 

Score: 

“5” 

“ 4 ” 

“3” 

“ 2 ” 

66 | 55 

Description: 

“100-91% 

compliance” 

“90-81% 

compliance” 

“80-71% 

compliance” 

“70-61% 

compliance” 

“60% or less 
compliance” 


Table 4. Choices of Score-Answers for Question #2 
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1. Purpose of Question #2: 

Question #2 requests the stakeholder’s assessment of the level of compliance 
observed for the COMOPTEVFOR instruction and expects some supporting evidence for 
the assessment. The answer to this question provides the basis for comparing the 
practices of VV&A under the COMOPTEVFORINST standard and judging success. 

2. Scores Observed for Question #2: 

Table 5 shows the average score-answers to Question #2. The overall average 
score is 3.9, which indicates that the typical stakeholder surveyed sees “90-81% 
compliance” with the instruction. Those performing the Tester/Accreditor role generally 
answered “80-71% compliance” and scored a 3.2 average, the lowest score among the 
stakeholder role categories. On the other hand, the stakeholder representing the VV&A 
Standards Official role provided a score of 5, the highest score among the groups, 
indicating that he/she thinks that “100-91% compliance” was achieved for the instruction. 
Those performing the Program Manager role scored a 4.4 average (which is 0.5 above the 
overall average), indicating that they typically see at least “90-81% compliance” with the 
instruction. The Resource Sponsors scored an average of 3.7, which is just 0.2 below the 
overall average. The average scores within the lower, middle, and upper management 
levels are 4.6, 3.5, and 3.4 respectively. These scores indicate that stakeholders at the 
lower management level see a noticeably higher level of compliance (at “100-91% 
compliance”) with the instruction, whereas stakeholders at the middle and upper 
management levels see only “90-81% compliance.” 

3. Stakeholder Comments for Question #2: 

The stakeholders’ comment-answers indicated that no consensus exists regarding 
the degree of compliance with the VV&A instruction. Instead, the comments suggest 
that there are two camps of experience. The first camp saw a fairly high level of 
compliance and pointed to a number of examples. One striking example was recounted 
by a Resource Sponsor, as follows: 
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Average Score-Answers to Question #2 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

5.0 

4.0 

1.0 

3.2 

Resource Sponsor 

5.0 

1.0 

5.0 

3.7 

Program Manager 

4.0 

4.5 

5.0 

4.4 

VV&A Standards Official 



5.0 

5.0 

Avg. Score within Level 

4.6 

3.5 

3.4 

3.9 


Table 5. Observed Level of Compliance with VV&A Instruction 


A recently completed VV&A effort enabled the elimination of a 
$70M physical T&E event required for a major ship acquisition. 
This was achieved because the PM employed a “test-model-test” 
iterative effort over a number of years in the ship’s RDT&E 
program to collectively understand the physics of the event on the 
ship. Further, the ship’s designs, tests, and models were planned to 
work in support of each other. The ship’s subassembly 
construction techniques facilitated M&S and VV&A. Finally, 
M&S process control was achieved to at least CMMI level 3 
standards. This facilitated a rapid VV&A process. 


Included in this first camp was an acknowledgement by many that, though overall 

compliance was high, there was less success at delivering the V&V products on time. 

(The stakeholders went on to say that the lateness of documentation was usually not 

critical to the process.) On the other hand, the second (though smaller) camp perceived a 

low level of compliance by the M&S proponent, citing cases such as this: 

... the Resource Sponsor ran into a big issue at the end of one 
VV&A process: DOT&E was not satisfied with the capability of 
the M&S to credibly predict the performance of the weapon system 
modeled. The Resource Sponsor wanted to use M&S to avoid the 
cost and risk entailed in conducting a certain close-in live-fire 
testing event. But the PM had not engaged the Tester/Accreditor 
sufficiently (or early enough) in their VV&A process to identify 
the shortcomings of the M&S, such that DOT&E rejected the use 
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of the M&S for the close-in event. Consequently, the Resource 
Sponsor had to reprogram $250M to conduct a physical test that 
the M&S was hoping to avoid. 


This second camp also observed that “many PMs did not seem to be knowledgeable 
about the instruction.” 

Beyond the individual cases mentioned, both camps offered reasons why 

proponents, when highly compliant, were successful. Reasons for success included: “The 

PMs used COMOPTEVFOR’s Surface Warfare VV&A Guide;” and “The PMs used the 

process diagram developed by NDAT.” One particularly noteworthy comment on the 

path to successful compliance was offered, as follows: 

The maturity and experience of the team that develops the ship’s 
T&E program, which includes M&S in its mix, has a lot to do with 
any successful compliance with the VV&A instruction. 

Finally, it was observed that the PMs generally had good intentions to comply 
with the instruction, as demonstrated by their efforts to learn the process through 
attendance at VV&A forums and working with NMSO to understand the mechanics of 
the VV&A process, including standard documentation requirements. 

4. Score / Comment Associations for Question #2: 

Table 6 shows that stakeholders’ scores fall into two distinct groups. The 
majority (11 of 14) of stakeholders gave scores of 5 and 4, indicating that a high level of 
instruction compliance was experienced. On the other hand, a vocal minority (3 of 14) 
scored compliance at “1” and commented that some M&S proponents were deficient in at 
least one critical area of the VV&A process. “In one case, the PM did not provide the 
data to finish the VV&A package. So the data had to be developed by the OT himself.” 
The lack of “3” and “2” scores indicate that there appears to be no middle ground among 
the stakeholders regarding level of compliance observed. 


69 



Distribution of Question #2 Score-Answers 

Level of Compliance with VV&A Instruction 

Count 

Stakeholders who provided a score of “5” 

7 

Stakeholders who provided a score of “4” 

4 

Stakeholders who provided a score of “3” 

0 

Stakeholders who provided a score of “2” 

0 

Stakeholders who provided a score of “1” 

3 

Total 

14 


Table 6. Distribution of Question #2 Scores-Answers 


5. Conclusions for Question #2: 

With an overall stakeholder average score of 3.9 (out of a perfect 5.0) and noting 
that 11 out of 14 stakeholders gave scores of at least 4, it appears that most stakeholders 
think that there was a moderately high level of compliance with COMOPTEVFORINST 
5000.1 A. On the other hand, a vocal minority felt strongly that there were clear instances 
of non-compliance and provided evidence and rationale for these observations. Some 
non-compliance issues were also noted within the overall positive comments of the 
majority (e.g., the high level of compliance achieved with respect to “content,” though 
not as high with respect to “schedule”). Some stakeholders expressed complex scores, 
which indicated positives for some areas of VV&A performance and admitted negatives 
for others. In general, it appears that stakeholders frankly expressed their opinions and 
freely pointed to discrepancies. Thereby is indicated that there is room for improvement 
in the duty of complying with the COMOPTEVFOR VV&A instruction. 

E. ANALYSIS OF QUESTION #3 

Question #3 asked: “What do you see to be the benefits associated with 
compliance with this instruction?” 
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Table 7 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of benefits seen in complying with the 
instruction. 


Choices of Score-Answers for Question #3 

Score: 

“5” 

“ 4 ” 

“3” 

“ 2 ” 

66 59 

Description: 

“many 

significant 

benefits” 

“several 

significant 

benefits” 

“one 

significant 

benefit” 

“a few 
minor 
benefits” 

“no 

real 

benefit” 


Table 7. Choices of Score-Answers for Question #3 


1. Purpose of Question #3: 

Question #3 solicits the stakeholder’s perception of the benefits associated with 
successfully complying with the COMOPTEVFOR instruction and expects some 
supporting examples for the assessment. The answer to this question is likely to provide 
the most compelling reasons for supporting the VV&A process advocated by 
COMOPTEVFORINST 5000.1 A. 

2. Scores Observed for Question #3: 

Table 8 shows the average score-answers to Question #3. The overall average 
score is 4.5, which indicates that the typical stakeholder surveyed sees “many significant” 
benefits associated with compliance with this instruction. This strong positive on 
benefits is particularly seen for those who perform the Tester/Accreditor role and those at 
the Middle and Upper Management levels, who scored averages of 4.8, 4.8, and 4.6 
respectively (the highest scores among the stakeholder role categories). The lowest 
group score is 4.0, which comes from the VV&A Standards Official role, who sees 
“several significant” benefits in complying with the instruction. Program Managers, 
Resource Sponsors, and those at Lower Management levels scored averages of 4.4, 4.3, 
and 4.2 respectively. In sum, the scores were tightly grouped between 4.0 and 5.0, 
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indicating that all stakeholders perceive significant benefits, which range from “several” 


to “many.” 


Average Score-Answers to Question #3 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

5.0 

5.0 

4.5 

4.8 

Resource Sponsor 

4.0 

4.0 

5.0 

4.3 

Program Manager 

3.5 

5.0 

5.0 

4.4 

VV&A Standards Official 



4.0 

4.0 

Avg. Score within Level 

4.2 

4.8 

4.6 

4.5 


Table 8. Stakeholder Perceived Benefits of Compliance with VV&A Instruction 

3. Stakeholder Comments for Question #3: 

The stakeholders’ comment-answers indicated that a strong consensus exists 
regarding the high number and kind benefits of complying with the VV&A instruction. 
Every stakeholder provided examples that highlighted at least one significant benefit, and 
nine of 14 stakeholders mentioned two or more benefits. Eight benefits of compliance 
with COMOPTEVFORINST 5000.1 A cited were: 

a. Better assurance that M&S will meet the needs of the user. 

b. Higher quality products are delivered in a shorter time. 

c. Financial savings due to management efficiencies. 

d. Accreditation follows naturally from the proponent’s V&V effort. 

e. Avoids program delays in reaching Acquisition Milestone B. 

f. Avoids the need to reprogram RDT&E funds to cover more OT. 

g. Promotes standardization, commonality, and M&S reuse. 

h. M&S and OT problems are identified and defined early. 
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As one stakeholder summed up, “The bottom-line is that complying with the instruction 
successfully obtains the Accreditor’s acceptance of the use of the M&S.” 


4. Score / Comment Associations for Question #3: 

Table 9 shows that the stakeholders scored the benefits of compliance highly, 

where the single lowest score of “3” indicates at least “one significant benefit.” A super- 

majority (13 of 14) of the stakeholders saw “several to many significant benefits.” in 

compliance with instruction. One comment described a nexus of benefits realized from 

long-term program investment in M&S and adherence to COMOPTEVFOR’s process: 

One particular U.S. Navy weapons acquisition program made 
heavy use of M&S for tens of years. What is in place today has 
saved a lot of money. The program has end-to-end testing and 
M&S capability, which provides flexibility in using different 
modes, and uses an iterative model update process. The program 
is able to work a combination of modeling and simulation, where 
the simulation generates environmental inputs into real hardware 
so that the weapon’s mission can be run in a very realistic way. 


Distribution of Question #3 Score-Answers 

Level of Benefits Perceived with Compliance 

Count 

Stakeholders who provided a score of “5” 

8 

Stakeholders who provided a score of “4” 

5 

Stakeholders who provided a score of “3” 

1 

Stakeholders who provided a score of “2” 

0 

Stakeholders who provided a score of “1” 

0 

Total 

14 


Table 9. Distribution of Question #3 Scores-Answers 


5. Conclusions for Question #3: 

With an overall average score of 4.5 (out of 5.0) and a multitude of benefits cited, 
it is clear that a strong consensus exists among the VV&A stakeholder community that 
compliance with the COMOPTEYFOR instruction yields many significant benefits. The 


73 




most obvious benefit of compliance cited was the high likelihood of Tester/Accreditor 
acceptance of the proponent’s M&S for the use in addressing a COI under OT. The 
comments revealed that many other program benefits result from following the process 
that achieves this bottom-line benefit. In large part, the stakeholders recognized that 
early interaction between the PM and COMOPTEVFOR was key to meeting OT 
requirements and avoiding any delays to reaching program milestones. Many saw overall 
program cost reduction and standardization of M&S products as significant benefits 
realized through compliance. That all stakeholders interviewed were readily able to 
identify some benefits of compliance provides reason to believe that COMOPTEVFOR’s 
instruction is basically successful. 


F. ANALYSIS OF QUESTION #4 

Question #4 asked: “What do you see to be the costs associated with compliance 
with this instruction?” 

Table 10 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of costs seen in complying with the 
instruction. 


Choices of Score-Answers for Question #4 

Score: 

“5” 

“ 4 ” 

“3” 

“ 2 ” 

(.(. 55 


“many 

“several 

“one 

“a few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 


costs” 

costs” 

cost” 

costs” 

cost” 


Table 10. Choices of Score-Answers for Question #4 


1. Purpose of Question #4: 

Question #4 solicits the stakeholder’s perception of the costs associated with 
obediently complying with the COMOPTEVFOR instruction and expects some 
supporting examples for the assessment. The answer to this question is likely to cite the 
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negative financial impacts of dutiful compliance with an instruction that leaves the issue 
open for the M&S proponent to address and the sponsor to support. 

2. Scores Observed for Question #4: 

Table 11 shows the average score-answers to Question #4. The overall average 
score is 2.4, which indicates that the typical stakeholder sees “a few minor costs” in 
complying with the instruction. But a further breakdown of the scores reveals a strong 
divergence of opinion between subcategories of stakeholder role and management level. 
Regarding those performing the Tester/Accreditor role, it would seem from their group 
average score of 2.6 that their general outlook on the cost of compliance falls somewhere 
between “a few minor costs” and “one significant cost.” However, this generalization 
would mask a clear difference of opinion within the management levels of that 
stakeholder role. The Tester/Accreditors are split between their members at the lower 
management level, who scored an average of 4.0 (thereby indicating “several significant 
costs”) and the organization’s middle and upper level managers, who scored averages of 
1.0 and 2.0 respectively (thereby indicating “no real cost” or “a few minor costs” 
respectively). Similarly, the Program Manager stakeholder group, which averaged a 2.8 
score, is divided into two factions. In this instance, the PM’s lower management level 
does not perceive more than “a few minor costs,” providing a group average score of 1.5. 
On the other hand, the middle and upper management levels of the PM group “several 
significant costs” with average scores of 3.5 and 4.0, respectively. In contrast, there is 
fair unanimity of opinion, “few to no costs” among the Resource Sponsors (1.7 average 
score) and also the VV&A Standards Official group (scoring 1.0), regardless of 
management level. 

3. Stakeholder Comments for Question #4: 

There was a divergence of opinions expressed among the stakeholders’ comment- 
answers. However, this disagreement can be grouped into two like-minded camps. The 
majority (9 of 14) of stakeholders felt that only a few minor costs (if any) were truly 
added to U.S. Navy ship acquisition programs’ costs of delivering M&S products useful 
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for OT. Two comments were typical of the majority opinion. The first point was made 

by a member of the Tester/Accreditor group, as follows: 

Ultimately, there are no genuine added costs associated with 
compliance. It could be argued, further, that the total cost to use 
M&S for OT&E is probably less than that which would be needed 
to perfonn physical testing. 


Average Score-Answers to Question #4 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

4.0 

1.0 

2.0 

2.6 

Resource Sponsor 

2.0 

2.0 

1.0 

1.7 

Program Manager 

1.5 

3.5 

4.0 

2.8 

VV&A Standards Official 



1.0 

1.0 

Avg. Score within Level 

2.6 

2.5 

2.0 

2.4 


Table 11. Stakeholder Perceived Costs of Compliance with VV&A Instruction 


The second such comment was offered by a Resource Sponsor, to wit: 

Compliance with the instruction means more “up-front time” and 
man-hours devoted to PM - COTF interactions than would the 
typical PM (in all likelihood) devote to VV&A conducted without 
the obligation of the instruction. However, without the instruction, 
the PM would end up having to collect more ‘real’ data (thereby 
incurring more cost) to make the M&S credible. 


Comments provided by minority, on the other hand, pointed out the most palpable of the 

costs incurred by the PM, such as this concern: 

The stringent documentation and configuration management 
requirements demanded by the instruction will most likely increase 
costs for the developers. 

That said, it seems that minority still recognized the potential of COMOPTEVFOR’s 
VV&A process to avoid help avoid added RDT&E costs and actually save overall 
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acquisition costs. A good “take-away” quote that expressed this sentiment was offered 
by the VV&A Standards Official stakeholder: “Compliance with the instruction really 
provides a savings to the U.S. Navy.” 


4. Score / Comment Associations for Question #4: 

Review of Table 12 distribution of score-answers confirms that there appears to 
be two groups of opinion regarding the cost of compliance. The minority (5 of 14) group, 
seeing one to several significant costs (scores of 3 and 4), expressed concern about 
having to pay more to produce “nice-to-have” high-fidelity M&S and detailed VV&A 
documentation. On the other hand, the majority (9 of 14) saw only a few (if any) minor 
costs associated with compliance, but also admitted that performing VV&A incurs more 
“upfront costs.” As one stakeholder offered, “When it comes to compliance, there is no 
free lunch!” 


Distribution of Question #4 Score-Answers 

Level of Costs Perceived with Compliance 

Count 

Stakeholders who provided a score of “5” 

0 

Stakeholders who provided a score of “4” 

4 

Stakeholders who provided a score of “3” 

1 

Stakeholders who provided a score of “2” 

5 

Stakeholders who provided a score of “1” 

4 

Total 

14 


Table 12. Distribution of Question #4 Scores-Answers 


5. Conclusions for Question #4: 

The difference of opinion among stakeholders as to whether significant costs are 
incurred by the PM in diligently complying with COMOPTEVFORINST 5000.1 A 
indicates that some further changes may be needed in the way VV&A process is managed 
and its costs budgeted. Not all the stakeholders, who presumably work together at the 
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key points in the process, see eye to eye on the financial investments demanded by the 
process. It is also interesting to note that stakeholders from the lower management level 
of the Tester/Accreditor group, who would normally not be expected to be sensitive to 
costs, identified “several significant costs” in complying with the COMOPTEVFOR 
instruction. It may be that, by their closer working relationships with their M&S 
proponent counterparts, they “feel the pain” of the costs carried by the Program Managers 
as they undergo the VV&A process. Of course, the PMs at the middle and upper levels 
of management, who scored averages of 3.5 and 4.0, would be very much expected to be 
sensitive to the costs of compliance, since they must balance their programs’ budgets. 


G. ANALYSIS OF QUESTION #5 

Question #5 asked: “What do you see to be the risks associated with compliance 
with this instruction?” 

Table 13 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of risks seen in complying with the 
instruction. 


Choices of Score-Answers for Question #5 

Score: 

“ 5 ” 

“ 4 ” 

“3” 

“2 ” 

66 59 


“many 

“several 

“one 

“a few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 


risks” 

risks” 

risk” 

risks” 

risk” 


Table 13. Choices of Score-Answers for Question #5 


1. Purpose of Question #5: 

Question #5 solicits the stakeholder’s perception of the risks associated with 
appropriately complying with the COMOPTEVFOR instruction and expects some 
supporting examples for the assessment. The answer to this question is likely to offer up 
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concerns that stakeholders have while following the instruction - uncertainties in the 
process that would undermine final accreditation of the proposed M&S. 


2. Scores Observed for Question #5: 

Table 14 shows the average score-answers to Question #5. The overall average 
score is 1.7, which indicates that the typical stakeholder sees “a few minor risks” in 
compliance with the instruction. The average scores of the individual stakeholder roles 
held very close to the overall 1.7 average, with scores of 1.8, 1.7, 1.8, and 1.0 for the 
Tester/Accreditor, Resource Sponsor, PM, and VV&A Standard roles, respectively. 
Lower and middle management levels scored averages of 2.0 for every participating 
stakeholder role, except for the 1.5 average score observed for PM middle management. 
Upper management issued the lowest overall average score of 1.4, with individual 
average scores of 1.5, 1.0, 2.0, and 1.0 for the Tester/Accreditor, Resource Sponsor, 
Program Manager, and VV&A Standard roles, respectively, indicating that “no real risk” 
is taken by proponents in complying with the COMOPTEVFORINST 5000.1 A. 


Average Score-Answers to Question #5 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

2.0 

2.0 

1.5 

1.8 

Resource Sponsor 

2.0 

2.0 

1.0 

1.7 

Program Manager 

2.0 

1.5 

2.0 

1.8 

VV&A Standards Official 



1.0 

1.0 

Avg. Score within Level 

2.0 

1.8 

1.4 

1.7 


Table 14. Stakeholder Perceived Risks of Compliance with VV&A Instruction 

3. Stakeholder Comments for Question #5: 

The stakeholder comment-answers indicated that consensus was obtained on the 
matter of risk with regard to complying with the COMOPTEVFOR instruction. The vast 
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majority stakeholder comments observed that there were “few, if any,” real risks involved 

in the business of complying with the instruction. At the same time, most of their 

comments acknowledged that less substantial risks were present - just not as bad as first 

feared by the M&S proponent. This comment was typical: 

There are political risks, which are inherent in the business of VV 
&A. Compliance with the instruction makes some of the big risks 
go away, but at the same time introduces a few new, smaller ones. 


The notion of “inherent risk” echoed though the stakeholders’ comments, as shown by 
this statement: 


The greatest risk the PM could encounter by dutifully complying 
with the instruction would be to discover, through the M&S for 
OT&E process, that the system to be acquired does not deliver the 
required capability and consequently should not be fielded. But 
this is precisely what OT&E is intended to discover and thus an 
inherent risk (whether M&S or physical testing is used) for the PM 
who presents his system for OT&E. 


There was also this observation on the challenge of tackling difficult technical issues: 

VV&A planning should go hand in hand with engineering software 
development. No significant risk should arise with compliance. 
However, there are minor risks that come from the instruction’s 
effect of driving PMs to comply to processes that produce 
relatively ‘safe’ (less pioneering) outcomes rather than 
encouraging these PMs to tackle difficult technical issues that, 
while they may not lead to as high a percentage of compliance, still 
produce greater M&S capability. 


Dissenting opinions were offered by two stakeholders, who voiced the existence of at 

least “one significant risk,” one of which was described as follows: 

There is the risk of last minute changes to test requirements or 
objectives, which may cause delays in accreditation (and thus test 
schedule) because stepping back the process will require rework on 
many artifacts and add extra V&V activities. 


The overall effect of the stakeholders’ comments was to acknowledge the existences of 
risks but point out that they are very manageable within the context of conducting VV&A 
in accordance with COMOPTEVFOR-INST 5000.1 A. 
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4. Score / Comment Associations for Question #5: 

The distribution of individual stakeholder score-answers shown in Table 15 shows 
that all of the scores lay at the bottom of the assessment of risk scale. 12 of 14 
stakeholders said that there were few, if any, risks associated with VV&A instruction 
compliance. The remaining two cite “one significant risk” but, as discussed above, the 
risks are still viewed to be manageable. 

5. Conclusions for Question #5: 

The vast majority of stakeholders sampled viewed the risk of complying with the 
instruction as “small” and “well within the ability to manage.” Nevertheless, most 
stakeholders also acknowledged that “risk” still weighed on the mind of the M&S 
proponent. The comments sought to explain the context for managing these risks, 
describing them as “inherent” in the nonnal course of presenting M&S for accreditation. 


Distribution of Question #5 Score-Answers 

Level of Risks Perceived with Compliance 

Count 

Stakeholders who provided a score of “5” 

0 

Stakeholders who provided a score of “4” 

0 

Stakeholders who provided a score of “3” 

2 

Stakeholders who provided a score of “2” 

6 

Stakeholders who provided a score of “1” 

6 

Total 

14 


Table 15. Distribution of Question #5 Scores-Answers 


H. ANALYSIS OF QUESTION #6 

Question #6 asked: “What do you see to be the benefits associated with non- 
compliance with this instruction?” 

Table 16 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of benefits seen in not complying with 
the instruction. 


81 




Choices of Score-Answers for Question #6 

Score: 

“ 5 ” 

“ 4 ” 

“3” 

“ 2 ” 

55 

Description: 

“many 

significant 

benefits” 

“several 

significant 

benefits” 

“one 

significant 

benefit” 

“a few 
minor 
benefits” 

“no 

real 

benefit” 


Table 16. Choices of Score-Answers for Question #6 


1. Purpose of Question #6: 

Question #6 solicits the stakeholder’s perception of any benefits associated with 
not fully complying with the COMOPTEVFOR instruction and expects some supporting 
examples for the assessment. The answer to this question is likely to supply some 
evidence of the difficulties encountered in following the COMOPTEVFOR VV&A 
process, where non-compliance produces some seeming benefits. 

2. Scores Observed for Question #6: 

Table 17 shows the average score-answers to Question # 6 . The overall average 
score is 1 . 2 , which indicates that the typical stakeholder surveyed sees “no real benefit” 
with instruction non-compliance. Those performing the Tester/Accreditor role generally 
answered “no real benefit” but provided the highest average score of 1.4 among the 
stakeholder role categories. (Thus, some of the Tester/Accreditors saw “a few minor 
benefits.”). The average scores of the stakeholder roles only went down from here. 
Program Managers came next, with an average score of 1.2, followed by both the 
Resource Sponsors and the VV&A stakeholder groups, who both posted the lowest 
possible average scores of 1.0. Only in the lower management level of the stakeholders, 
who scored an average of 1 . 6 , was there any indication of benefit to non-compliance. 
This management level average was influenced by the 2.0 average of the 
Tester/Accreditors within that level, followed by Program Managers at 1.5, then 
Resource Sponsors at 1.0. All middle and upper level managers scored this question at 
1.0. In sum, the scores indicate very strong stakeholder agreement that no real benefits 
are obtained from non-compliance with COMOPTEVFOR 5000.1 A. 
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Average Score-Answers to Question #6 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

2.0 

1.0 

1.0 

1.4 

Resource Sponsor 

1.0 

1.0 

1.0 

1.0 

Program Manager 

1.5 

1.0 

1.0 

1.2 

VV&A Standards Official 



1.0 

1.0 

Avg. Score within Level 

1.6 

1.0 

1.0 

1.2 


Table 17. Stakeholder Perceived Benefits of Non-compliance w/ VV&A Instruction 

3. Stakeholder Comments for Question #6: 

The stakeholder comment-answers signified the strongest accord of opinion 
among the stakeholders for a given question. To a person, no stakeholder admitted of 
any real benefit to instruction non-compliance. Even those few that indicated scores of 
“2” explained in their comments that any benefit derived from non-compliance was really 
a perversity. This comment from a Test/Accreditor (at the lower management level) is a 
case in point: 

From the OT perspective, there may be (perversely) one significant 
benefit associated with non-compliance. In the event the PM has 
not complied, he has thus not justified his proposal to reduce the 
physical testing required for OT. This provides leverage for the 
OTD to require more testing to make up for the lack of support that 
accredited M&S provides. So, while non-compliance may save 
money on M&S at the front end, the PM ends up paying more for 
testing at the back end. Moreover, physical testing uncovers 
problems later when they cost more to fix (if they can be fixed at 
that point), which risks meeting fewer requirements. 

This comment makes abundantly clear that any perceived “benefit” of non-compliance is 
really a liability, since the proponent will end up spending more and waiting longer to 
appropriately address the required COI of OT and thus satisfy the Tester/Accreditor. As 
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a point of departure, the issue of what is meant by “non-compliance” versus “tailoring” of 

the VV&A process came to light in one of the PM stakeholder’s comments, thusly: 

Not complying with COTF instruction would cause much more 
pain than benefit (if any). However, PMs should find ways to 
tailor the instruction to fit within the constraints of their acquisition 
programs without having a negative effect on the integrity of the 
VV&A process. 

The researchers noted from their literature reviews and attendance at the VV&A Forums 
that tailoring is proposed as a technique that enables PMs to individually comply with the 
prescribed VV&A process in accordance with the particular circumstances / constraints 
of their programs. However, no further interview comments were provided in this area. 


4. Score / Comment Associations for Question #6: 

The distribution of individual stakeholder score-answers shown in Table 18 shows 
that all of the scores lay at the bottom of the assessment of risk scale. 12 of 14 
stakeholders said that there was “no real benefit” with instruction non-compliance. The 
remaining two are distributed at the “2” and “3” score levels, principally to admit that 
some “perverse” benefits are derived but, as discussed above, are really bigger liabilities. 
The idea of “tailoring” as a mechanism for PMs to vary their approach to VV&A process 
compliance was raised but did not go further in discussion. 


5. Conclusions for Question #6: 

The idea of reaping any benefit through non-compliance with COMOPTEVFOR- 
INST 5000.1 A was really a “non-starter” for most of the stakeholders interviewed. One 
stakeholder opined, “This question doesn’t really make sense, because non-compliance 
with instruction brings no benefit whatsoever.” Nevertheless, asking the question was 
useful, as it generated some interesting comments on the adverse impacts of non- 
compliance. These included: (a) “M&S activities are not coordinated with acquisition 
strategy;” (b) “T&E strategy does not provide validation data;” (c) “Models are not 
accredited prior to use;” and (d) “M&S use and VV&A strategy are not in the TEMP.” 
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The question also raised the issue of “tailoring” the VV&A process to accommodate the 
individual constraints of the various surface ship acquisition programs. 


Distribution of Question #6 Score-Answers 

Level of Benefits Perceived with Non-Compliance 

Count 

Stakeholders who provided a score of “5” 

0 

Stakeholders who provided a score of “4” 

0 

Stakeholders who provided a score of “3” 

1 

Stakeholders who provided a score of “2” 

1 

Stakeholders who provided a score of “1” 

12 

Total 

14 


Table 18. Distribution of Question #6 Scores-Answers 


I. ANALYSIS OF QUESTION #7 

Question # 7 asked: “What do you see to be the costs associated with non- 
compliance with this instruction?” 

Table 19 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of costs seen in not complying with the 
instruction. 


Choices of Score-Answers for Question #7 

Score: 

“5 ” 

“ 4 ” 

“3” 

“ 2 ” 

U 55 


“many 

“several 

“one 

“a few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 


costs” 

costs” 

cost” 

costs” 

cost” 


Table 19. Choices of Score-Answers for Question #7 
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1. Purpose of Question #7: 

Question #7 solicits the stakeholder’s perception of the costs associated with not 
complying with the COMOPTEVFOR instruction and expects some supporting examples 
for the assessment. The answer to this question is likely to point to momentous financial 
dangers of non-compliance, which ultimately must be controlled by the M&S proponent 
and reconciled by the sponsor. 

2. Scores Observed for Question #7: 

Table 20 shows the average score-answers to Question #7. The overall average 
score is 4.0, which indicates that the typical stakeholder surveyed sees “several 
significant costs” incurred when not complying with the instruction. Those perfonning 
the Tester/Accreditor role generally answered “many significant costs” and scored a 4.6 
average, the highest score among the stakeholder role categories. Next in score order was 
the VV&A Standards Official role, who provided a score of 4, which exactly matched the 
overall stakeholder average, likewise indicating “several significant costs.” Those 
performing the Resource Sponsor role scored a 3.7 average (just 0.3 below the overall 
average), also indicating that “several significant costs” would be suffered through non- 
compliance with the instruction. The Program Managers scored an average of 3.6, which 
is still just 0.4 below the overall average and describes the same cost impact. The 
average scores within the lower, middle, and upper management levels are 4.6, 4.0, and 
3.4 respectively. These scores indicate that stakeholders at the three management levels 
vary +/- 0.6 around the mean of “several significant costs” regarding non-compliance 
with the instruction. Thus, the relatively small variance of stakeholder role and 
management level average scores indicates a strong consensus of all stakeholders that the 
costs of non-compliance are both “significant” and “several.” 

3. Stakeholder Comments for Question #7: 

The stakeholders’ comment-answers indicated that a strong consensus exists 
regarding the high costs of non-compliance with the VV&A instruction. The keynote of 
this hannony was heard from the very first Tester/Accreditor interviewed, as follows: 
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The proponent cannot perfonn adequately the needed design 
studies, trade studies, pre-test planning, pre-test predictions, and 
post-test analysis when the M&S is of poor quality, as would be 
indicated by its failure to be accredited. One cannot trust the 
results! Consequently, it is harder to analyze and solve problems 
early. This failure would be avoided if sufficient funding were 
applied to M&S development and its supporting VV&A. While 
the costs associated with non-compliance are sometimes hard to 
quantify, the effects are nonetheless apparent. 


Average Score-Answers to Question #7 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

5.0 

5.0 

4.0 

4.6 

Resource Sponsor 

4.0 

3.0 

4.0 

3.7 

Program Manager 

4.5 

4.0 

1.0 

3.6 

VV&A Standards Official 



4.0 

4.0 

Avg. Score within Level 

4.6 

4.0 

3.4 

4.0 


Table 20. Stakeholder Perceived Costs of Non-compliance with VV&A Instruction 


The other stakeholders sang essentially the same song in their comments. One 

particularly strong solo, which offered a stark example of a cost consequence, was given 

by a Resource Sponsor, as follows: 

In the case of one ship acquisition program, it cost $250M (which 
had to be taken from other program funds) to perform the physical 
operational test that the M&S sought to accomplish in a virtual 
environment. Perhaps earlier PM interaction with the Tester/ 
Accreditor and the collection of some additional physical (real) 
data would have led to a credible M&S solution. 


Program Managers, too, cited significant financial impacts for non-compliance. This 

thoughtful comment particularly resonated: 

Not following the VV&A instruction creates a significant 
probability of incurring added costs, especially when trying to 
validate and verify the models of federations. On what basis will 
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PMs be confident that the ship systems they are delivering will 
perforin the way they are predicting? Physical testing will have to 
be added to prove the VV&A case. This is necessary because no 
one can buy or accept a product of which they are unsure, 
especially when the stakes are high. The costs are ultimately borne 
by the warfighter. 


Finally, the VV&A Standards Official role voiced this final note: “Non-compliance with 
the instruction would continue to cost the U.S. Navy more money.” 


4. Score / Comment Associations for Question #7: 

Table 21 shows from its distribution of raw scores that most (11 out of 14) 
stakeholders believe the costs incurred by not complying with COMOPTEVFORINST 
5000.1 A to be (at least) “several significant costs” (with scores of “4” and “5”). Two 
stakeholders saw at least “one significant cost” and, because of their attribution of 
“significant” with regard to costs experienced, should also be grouped with the super 
majority. Only one (lonely) dissenter is evident. This member of the PM group pointed 
out that, “There is no real cost associated with non-compliance if VV&A is not done. 
However, the cost to go back later and conduct VV&A will be much more expensive.” 
Thus, the one dissenting opinion is really saying (perversely) that non-compliance cost 
nothing if one chooses not to participate. 


Distribution of Question #7 Score-Answers 

Level of Costs Perceived with Non-Compliance 

Count 

Stakeholders who provided a score of “5” 

5 

Stakeholders who provided a score of “4” 

6 

Stakeholders who provided a score of “3” 

2 

Stakeholders who provided a score of “2” 

0 

Stakeholders who provided a score of “1” 

1 

Total 

14 


Table 21. Distribution of Question #7 Scores-Answers 
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5. Conclusions for Question #7: 

The vast majority of stakeholders interviewed consider the costs of not complying 
with the COMOPTEVFOR VV&A instruction to be (at least) “several” and “significant.” 
A number of concrete (i.e., real dollar value) examples were provided. Regardless of 
stakeholder role or management level in their respective organizations, all participants 
appreciated the potential cost increases to U.S. Navy surface ship acquisition programs if 
COMOPTEVFORINST 5000.1 A were not followed in support of M&S for OT 
accreditation. 

J. ANALYSIS OF QUESTION #8 

Question #8 asks: “What do you see to be the risks associated with non- 
compliance with this instruction?” 

Table 22 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of risks seen in not complying with the 
instruction. 


1. Purpose of Question #8: 

Question #8 solicits the stakeholder’s perception of the risks associated with not 
fully complying with the COMOPTEVFOR instruction and expects some supporting 
examples for the assessment. The answer to this question is likely to point out the 
hazards of deviating from the standards of VV&A practice put forth by the instruction 
and remind practitioners that the ultimate accreditation of the proposed M&S is at stake. 


Choices of Score-Answers for Question #8 

Score: 

“5” 

“ 4 ” 

“3” 

“2 ” 

U 55 


“many 

“several 

“one 

“a few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 


risks” 

risks” 

risk” 

risks” 

risk” 


Table 22. Choices of Score-Answers for Question #8 
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2. Scores Observed for Question #8: 

Table 23 shows the average score-answers to Question #8. The overall average 
score is 4.4, which indicates that the typical stakeholder surveyed sees (at least) “several 
significant risks” incurred when not complying with the instruction. Those perfonning 
the Program Manager generally answered “many significant risks” and scored a 4.8 
average, the highest score among the stakeholder role categories. Next in score order was 
the Tester/Accreditor role, who provided an average score of 4.4, which exactly matched 
the overall stakeholder average, likewise indicating (at least) “several significant risks.” 
The VV&A Standards Official role scored 4.0, which is still just 0.4 below the overall 
average and describes the same cost impact. Those performing the Resource Sponsor 
role scored a 3.7 average (0.7 below the overall average), which closely approaches the 
indication of “several significant risks” that would be suffered through non-compliance 
with the instruction. The average scores within the lower, middle, and upper 
management levels are 4.4, 4.5, and 4.2 respectively. These scores indicate that 
stakeholders at the three management levels vary +/- 0.3 (maximum) around the mean of 
“several significant risks” regarding non-compliance with the instruction. Thus, the 
relatively small variance of stakeholder role and management level average scores 
indicates a strong consensus among the stakeholders that non-compliance entails “several 
significant risks.” 


Average Score-Answers to Question #8 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

4.5 

5.0 

4.0 

4.4 

Resource Sponsor 

4.0 

3.0 

4.0 

3.7 

Program Manager 

4.5 

5.0 

5.0 

4.8 

VV&A Standards Official 



4.0 

4.0 

Avg. Score within Level 

4.4 

4.5 

4.2 

4.4 


Table 23. Stakeholder Perceived Risks of Non-compliance with VV&A Instruction 
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3. Stakeholder Comments for Question #8: 

The stakeholders’ comment-answers indicated that a strong consensus exists 

regarding the high risks of non-compliance with the VV&A instruction. The comments 

that identify the risks entailed support the sense of “real,” “varied,” and “significant.” 

The main point raised was captured in this stakeholder statement, “The risks are that the 

requirements of the M&S will not be met and therefore the OT&E test requirements 

would not be fully satisfied.” Another Tester/Accreditor offered this “real life” example: 

One significant risk associated with non-compliance was recently 
made evident where a ship acquisition program was held up at MS 
B while resolving the issue of whether the use of M&S would 
satisfy one of the OT requirements. 


A further case-in-point was made by a Resource Sponsor, as follows: 

Several significant risks are associated with non-compliant VV&A, 
including the need to “re-synchronize” or “re-coordinate” the 
acquisition program’s “tracks” (i.e., the kind of activities/events 
identified in the TEMP). As this would translate to cost increases, 
the PM would have to go back to the Resource Sponsor for help. 
But the Resource Sponsor’s solution is usually to advise the PM to 
“take it out of hide” and reprogram existing funds to support the 
changes. 


Finally, a PM provided this discussion on the risks of getting “false-positives,” to wit: 

Getting “false-positives” are the risks associated with non- 
compliance. It can be presumed that the PMs are moving forward 
in good faith to conduct VV&A on their M&S. However, when 
they do not use the COTF prescribed process, they end up having 
to develop their own. This will result in unevenness of results 
across the PMs. Some PMs will be more stringent than COTF, but 
most will likely be less stringent. Those who are less stringent run 
the risk of getting false-positive results, that is, of qualifying an 
M&S that is really deficient. The effects of that false positive will 
be rolled-up into the system to which the M&S is applied. The 
resulting disconnect between expected perfonnance and actual 
performance means “bad news” for the fleet. 


Many other observations and examples were cited. The VV&A Standards Official role 
(once again) made an excellent closing comment on the issue by stating: 
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There are several significant risks associated with non-compliance; 
the foremost among them is the lack of credibility for those models 
developed without regard to compliance. 

4. Score / Comment Associations for Question #8: 

Table 24 shows from its distribution of raw scores that virtually all of the 
stakeholders interviewed believe that, by not complying with COMOPTEVFORINST 
5000.1 A, M&S proponents would lay themselves open to (at least) “several significant 
risks” (thereby giving scores of “4” and “5”). The remaining stakeholder still thinks that 
the incurred risk is significant (but just cited one example). Other than the various 
examples of risk cited in their comments, many of the stakeholders noted that risks, when 
instantiated, most often translate into added program costs. 


Distribution of Question #8 Score-Answers 

Level of Risks Perceived with Non-Compliance 

Count 

Stakeholders who provided a score of “5” 

6 

Stakeholders who provided a score of “4” 

7 

Stakeholders who provided a score of “3” 

1 

Stakeholders who provided a score of “2” 

0 

Stakeholders who provided a score of “1” 

0 

Total 

14 


Table 24. Distribution of Question #8 Scores-Answers 


5. Conclusions for Question #8: 

An overwhelming majority of the stakeholders view the risks of not complying 
with the COMOPTEVFOR VV&A instruction to be (at least) “several” and “significant.” 
A number of “real life” examples were provided, which demonstrated the seriousness of 
the risks that non-compliant M&S proponents would take on. Risks realized are often 
“felt” in the form of added costs and extended schedules. The consensus demonstrated 
by the stakeholders signals that COMOPTEVFORINST 5000.1 A is effectively educating 
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the diverse U.S. Navy surface ship acquisition participants on the importance of 
following the standards of the VV&A process. 

K. ANALYSIS OF QUESTION #9 

Question # 9 asks: “To what extent did the change from the superseded 
COMOPTEVFORINST 5000.1 (of 5 September 1995) to the new COMOPTEVFOR- 
INST 5000.1 A (of 9 September 2004) improve or degrade the VV&A of M&S used for 
OT&E?” 

Table 25 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of improvement (or degradation) seen 
in changing to the new instruction. 


1. Purpose of Question #9: 

Question #9 is posed to elicit comment from the stakeholder that recognizes 
whether any improvements (or degradations) were obtained in replacing the old 
COMOPTEVFOR VV&A instruction with the new (current) instruction. This may not 
be an easy question to answer, as the stakeholder is expected to be sufficiently 
experienced with (or knowledgeable of) the letter, spirit, and practice of both instructions. 


Choices of Score-Answers for Question #9 

Score: 

“5” 

“ 4 ” 

“3 ” 

“2 ” 

66 55 

Description: 

“significant 

improve¬ 

ments” 

“minor 

improve¬ 

ments” 

“no real 
improve¬ 
ments” 

“minor 

degrada¬ 

tions” 

“significant 

degrada¬ 

tions” 


Table 25. Choices of Score-Answers for Question #9 
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2. Scores Observed for Question #9: 

Table 26 shows the average score-answers to Question #9. The overall average 
score is 3.9, which indicates that the typical stakeholder surveyed saw “minor 
improvements” in going to the new the instruction. 


Average Score-Answers to Question #9 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

3.0 

5.0 

5.0 

4.2 

Resource Sponsor 

5.0 

3.0 

3.0 

3.7 

Program Manager 

3.5 

3.5 

4.0 

3.6 

VV&A Standards Official 



4.0 

4.0 

Avg. Score within Level 

3.6 

3.8 

4.2 

3.9 


Table 26. Stakeholder Assessment of Improvement Gained with New Instruction 

Those performing the Tester/ Accreditor role generally answered “minor 
improvements” and scored a 4.2 average, the highest score among the stakeholder role 
categories. Next in order of score is the VV&A Standards Official role, who provided a 
score of 4, which is only 0.1 above the overall stakeholder average, indicating that he/she 
thinks that “minor improvements” were achieved by the new instruction. Then the 
Resource Sponsor and the Program Manager roles follow with average scores of 3.7 and 
3.6 (which are 0.2 and 0.3 below the overall average), respectively. These scores also 
indicate that the stakeholders saw “minor improvements” in the new instruction. The 
average scores within the lower, middle, and upper management levels are 3.6, 3.8, and 
4.2, respectively, which show a modest upward trend by the ascending levels of 
management in grading the levels of improvement found in the new instruction. 
Nevertheless, the scores indicate that all three management levels are saying “minor 
improvements” were achieved in moving from the old to the new instruction. 
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3. Stakeholder Comments for Question #9: 

The stakeholders’ comment-answers indicated that no consensus exists regarding 
the recognition of VV&A process improvements (or degradations) in moving from the 
old COMOPTEVFORINST 5000.1 to the new COMOPTEVFORINST 5000.1 A. Rather, 
the comments indicated that three perspectives are maintained: (a) No real improvement 
was achieved; (b) minor improvements were achieved; and (c) significant improvements 
were achieved. In the first case, a plurality of six (of the 14) stakeholders commented 
that no real improvements were evident in the instruction revision. The general remark 
of this group was, “The jury is still out.” They observed that the new instruction has not 
been in place long enough to collect data and perform a genuine assessment of its 
performance. Now the second group (numbering four stakeholders) has apparently seen 
some data that leads their members to conclude that “minor improvements” were 
achieved. Three points were made in support of this assessment: 

• The new instruction is more user-friendly and shorter. 

• The new instruction provides a sense of structure, format and process. [It] 
guides the PM through the steps of the VV&A process with only minor difficulties. 

• The new instruction better articulates accepted VV&A standards and 
provides more specific guidance than the general philosophy of the old instruction. 

Finally, the third group (also numbering four members) sees evidence of “significant 
improvement” in the move to the new instruction. Observations, such as these, were 
offered in support of this group’s view: 

• With the old instruction, PMs did not know what the requirements were. 

• [The new instruction] provides a far better roadmap than before. 

• The earlier the OTD is involved, the better the compliance. The new 
instruction better promotes this. 

• The changes incorporated in the new instruction are “golden,” as they 
allow the PM more flexibility while still expecting him to interact with COTF early in the 
process for delivering VV&A products. 
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Hence, the interviewed stakeholders’ statements demonstrate a three-way split in the 
community of VV&A participants regarding the level of improvement achieved by the 
recent revision of the COMOPTEYFOR instruction. 


4. Score / Comment Associations for Question #9: 

Table 27 shows from its distribution of raw scores that a three-way split exists 
regarding stakeholder opinion on the level of improvement gained in going to the new 
instruction, COMOPTEVFORINST 5000.1 A of 9 September 2004. The largest of the 
groups (with 6 members) graded the new instruction at “3,” which indicated their 
judgment that it caused (so far) “no real improvements” in the VV&A process practiced. 


Distribution of Question #9 Score-Answers 

Level of Improvement Gained with New Instruction 

Count 

Stakeholders who provided a score of “5” 

4 

Stakeholders who provided a score of “4” 

4 

Stakeholders who provided a score of “3” 

6 

Stakeholders who provided a score of “2” 

0 

Stakeholders who provided a score of “1” 

0 

Total 

14 


Table 27. Distribution of Question #9 Scores-Answers 


The remaining two groups comprised four members each, where the second camp saw 

“minor improvements” and the third cited “significant improvements.” These two 

factions each provided several observations to support their assessments. One score / 

comment association stood out in particular, indicating a shift in VV&A process focus: 

[A significant improvement] includes the insight that the 
participants (within the established accreditation process) should 
move from a focus on the fidelity of the model to the 
appropriateness of the model for its intended use. The level of 
resolution to be achieved for the model depends on the questions 
being asked. 
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Since the majority of the stakeholders’ W&A experiences were under the old 
instruction, the researchers found the answers to this question illuminating. 


5. Conclusions for Question #9: 

With an overall stakeholder average score of 3.9, but showing a three-way 
divergence of raw score-answers (as was also evident in the comment-answers), the 
answer to the question of, “Whether improvement was obtained by the new instruction?” 
boils down to the realization: “It depends.” The experience of the stakeholders varied on 
this issue but did not depend on the stakeholder role or management level. Therefore, it 
appears that the issue varies for the moment on the experiences of the individuals. 

L. ANALYSIS OF QUESTION #10 

Question #10 asks: “What improvements would you recommend for COMOP- 
TEVFOR’s Accreditation process (beyond that prescribed by the new COMOPTEVFOR- 
INST 5000.1 A)? 

Table 28 shows the selection of scores that each stakeholder was able to choose 
from in answering the question, assigning a level of opportunity for improvement beyond 
that achieved by the new instruction. 

1. Purpose of Question #10: 

Question #10 is posed to elicit recommendations for improvements to any future 
edition of the COMOPTEVFOR VV&A instruction or for improvements to the U.S. 
Navy’s VV&A process in general. While the answer to this question is speculative, it 
nevertheless provides an outlet for expressing solutions to the difficulties (regrettable 
costs and risks) still found in complying with the current COMOPTEVFOR VV&A 
instruction. 


97 



Choices of Score-Answers for Question #10 

Score: 

“5” 

“ 4 ” 

“3” 

“ 2 ” 

tt ^ 59 


“many 

“several 

“one 

“a few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

improve- 

improve- 

improve- 

improve- 

improve- 


ments” 

ments” 

ment” 

ments” 

ment” 


Table 28. Choices of Score-Answers for Question #10 

2. Scores Observed for Question #10: 

Table 29 shows the average score-answers to Question #10. The overall average 
score is 2.9, which indicates that the typical stakeholder surveyed saw an opportunity to 
make “one significant improvement” beyond that which is currently described in 
COMOPTEVFORINST 5000.1 A. Those performing the Tester/Accreditor role 

generally answered “one significant improvement” and scored a 3.2 average, the highest 
score among the stakeholder role categories. Next in order of score is the Resource 
Sponsor role, who provided an average score of 3.0, which is just 0.1 above the overall 
stakeholder average, indicating that this group thinks that “one significant improvement” 
can be made to the 5000.1 A instruction. Closely following is the Program Manager role, 
with an average score of 2.8 (just 0.1 below the overall average), likewise indicating that 
these stakeholders saw the opportunity for “one significant improvement” in the 5000.1 A 
instruction. Finally, well down in the score order, is the W&A Standards Official role, 
who provided a score of 1 , which indicates that he/she saw “no [opportunity for] real 
improvement.” The average scores within the lower, middle, and upper management 
levels are 3.0, 2.3, and 3.2, respectively, which shows a one point downward dip in the 
score trend at the middle management level. This indicates that mid-management, on 
average, only sees opportunities for “a few minor improvements,” while the lower and 
upper levels see “one significant improvement,” which closely corresponds to the overall 
stakeholder average. 
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Average Score-Answers to Question #10 

Stakeholder Role 

Management Level of Stakeholder 

Avg. Score 
within Role 

Lower 

Middle 

Upper 

T ester/Accreditor 

4.5 

1.0 

3.0 

3.2 

Resource Sponsor 

1.0 

3.0 

5.0 

3.0 

Program Manager 

2.5 

2.5 

4.0 

2.8 

VV&A Standards Official 



1.0 

1.0 

Avg. Score within Level 

3.0 

2.3 

3.2 

2.9 


Table 29. Perceived Opportunity for Improvement of COTF’s Accreditation Process 


3. Stakeholder Comments for Question #10: 

The stakeholders’ comment-answers indicated that no consensus exists regarding 

the opportunity for making improvements to the VV&A process described in the new 

COMOPTEVFORINST 5000.1 A. Rather, the comments indicated that there are two 

opposing camps on the issue (and possibly a third one somewhere in the middle). The 

two opposing perspectives are: (a) There is no real opportunity (or impetus) for 

improvement; and (b) There is opportunity for making significant improvements to the 

latest instruction. Six (of the 14) stakeholders make up the first camp and either 

recommended “no real improvement” or only “a few minor improvements” regarding 

COMOPTEVFORINST 5000.1 A. An identically sized (six of 14) second camp 

recommended either “many” or “several significant improvements” to the instruction. 

There are two stakeholders in the middle of the debate who see opportunity for only “one 

significant improvement.” However, this “middle ground” can really be assigned to 

Camp #2 (who recommend more than one significant improvement) because the single 

improvement they are recommending is still “significant,” as opposed to minor. The 

below comment from a Tester/Accreditor expresses the second camp’s view succinctly: 

Since OT is the final step before full production, it would be most 
efficient to offer the PM a VV&A instruction that contains both 
policy and detailed “how to” guidance a single, usable resource. 

The PM would there find efficient methods to organize, estimate 
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costs, determine schedules, and finish a quality M&S on time and 
within budget. 

A more detailed comment from second Tester/Accreditor addresses the need to revise the 

instruction to better support the reuse of M&S: 

Among the possible improvements to be made is finding a way to 
make models more available for reuse - both within the U.S. Navy 
and among all the Military Services. Testing in the Joint Force 
Environment relies on a having an adequately modeled virtual 
environment. More funding is needed to fully support this 
initiative, along with a technical commitment at and above any 
particular Service’s ability. Then the question can be asked, “What 
does a PM expect to be able to do when he/she wants to reuse the 
M&S developed by another, only to find that this PM is not 
following the Service’s VV&A process?” 


The opposing camp offered comments like this one: 

There are no real improvements that can be made to COTF’s 
Accreditation process. The key issue for improving the 
accreditation process regards the implementation of VV&A and 
not necessarily what the VV&A requires by way of procedure. 
Instead of focusing on the process for producing documentation, 
the participants should focus on the technical content conveyed in 
the documentation. In other words, the instruction should not 
emphasize how to do conduct the W&A process or require close 
adherence to a checklist. Rather, the focus should be on the 
appropriateness of the model to answer the technical question 
related to its intended use. 


The comment-answers generated by this question were the most varied and extensive of 
those collected by the interviews. The reader is encouraged to review the Appendix for 
the remaining comments that provide specific recommendations for improvements, or 
reasons why improvements to COMOPTEFORINST 5000.1 A are not warranted. 


4. Score / Comment Associations for Question #10: 

Table 30 shows from its distribution of raw scores that the two highest score 
counts (at 4) are for the scores of “4” and “1,” which represent the polar opposite 
recommendations of “several significant improvements” and “no real improvement.” 
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The remaining counts are half (at 2) of those assigned to each of the two poles. These 
suggest the down steps on either side of the two primary opinions. 


Distribution of Question #10 Score-Answers 

Level of Opportunity for VV&A Improvement 

Count 

Stakeholders who provided a score of “5” 

2 

Stakeholders who provided a score of “4” 

4 

Stakeholders who provided a score of “3” 

2 

Stakeholders who provided a score of “2” 

2 

Stakeholders who provided a score of “1” 

4 

Total 

14 


Table 30. Distribution of Question #10 Scores-Answers 


5. Conclusions for Question #10: 

The overall stakeholder average score of 2.9 masks the presence of two distinct 
camps of opinion regarding the opportunity to make further improvements to COM- 
OPTEVFOR’s VV&A instruction. The first camp does not see the need to improve the 
instruction, although they generally advise other improvements (such as making better 
efforts to follow the instruction “as is”). On the other hand, the second camp sees plenty 
opportunity to make improvements to the instruction, mostly because they believe a 
revised instruction can provide more “how to” guidance. Despite how well VV&A is 
doing under the current instruction, a simple majority (8 of the 14 interviewed) sees the 
need to make at least “one significant improvement” to COMOPTEVFORINST 5000.1 A. 


M, CONCLUSION 

This chapter analyzed the answers supplied for the ten questions administered to 
U.S. Navy Surface Ship VV&A stakeholders through semi-structured interviews. The 
data was sorted, averaged, and grouped to identify areas of consensus or divergence. 
Stakeholder comments were highlighted to assist the reader’s understanding of the issues 
raised and to present options for consideration. A conclusion was reached for each 
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question’s concern. This will provide the basis for making the global conclusions and 
recommendations that are offered in Chapter IV. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

This chapter provides conclusions that address the thesis objectives and answer 
the research questions that were posed in Chapter I. Recommendations are made for 
future studies. 

B. CONCLUSIONS 

The conclusions of this thesis summarize the responses of the four stakeholder 
groups - Tester/Accreditor, Resource Sponsor, Program Manager, and VV&A Standards 
Official - regarding their perspectives on the extent of, barriers to, and factors for PM 
compliance with COMOPTEVFORINST 5000.1 A, and possible improvements that can 
be made to enhance the instruction. 

1. Extent of Compliance with COMOPTEVFORINST 5000.1A 

The extent of compliance with COMOPTEVFORINST 5000.1 A is most clearly 
indicated by the stakeholders’ answers to the question: What level of compliance with the 
instruction do you think was achieved? 4 11 of 14 stakeholders interviewed judged 
compliance to be a “4 out of 5” toward full compliance. Thus, most of the stakeholders 
perceived a moderately high level of compliance with the instruction. Nevertheless, a 
noteworthy minority saw some clear instances of non-compliance and provided evidence 
and rationale for these observations. Non-compliance issues were also noted within the 
overall positive comments of the majority. Several stakeholders provided complex 
evaluations, which indicated positives for some areas of VV&A perfonnance and pointed 
to negatives for others. With this data, it can be reasonably concluded that the level of 
PM compliance with COMOPTEVFOR’s VV&A process is good but could be better. 


4 This was asked in semi-structured interview Question #2. Support for the reliability of the appraisals 
provided by Question #2 came from the answers to Question #1, which provided evidence of wide-ranging 
stakeholder VV&A experiences and working familiarity with COMOPTEVFORINST 5000.1A. 
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2. Barriers to Compliance with COMOPTEVFORINST 5000.1A 

The barriers to compliance with COMOPTEVFORINST 5000.1 A are indicated 
by the stakeholders’ answers to the three interview questions 5 that dealt with the “costs” 
of compliance, the “risks” of compliance, and the “benefits” of non -compliance. 

The first type of barrier to full compliance with COMOPTEVFOR’s VV&A 
process is the perceived “significant” costs borne by PMs. Four of the stakeholders 
interviewed with Question #4 asserted the concern for costs. One minority comment in 
particular pointed out that cost increases for M&S budgets are likely because of the 
“stringent documentation and configuration management requirements demanded by the 
instruction.” Sympathetically, a number of the stakeholders agreed that there were more 
“upfront” costs associated with compliance, particularly in the areas of M&S 
development and VV&A activities, and especially for new programs. However, the 
majority of stakeholders viewed examples like this to be “false” barriers because they 
judged that PMs would have to pay more over the life of their RDT&E programs for 
VV&A process non-compliance. 

Risk was perceived to be another (second) type of barrier to compliance, though 
most stakeholders who answered Question #5 found it to be only a minor problem. But 
one stakeholder saw a significant risk in complying with this instruction, should there be 
any last minute changes to test requirements/objectives. This stakeholder’s concern was 
that such changes could cause delays in accreditation and test schedules, “because 
stepping back would require rework on many artifacts, thus adding extra VV&A 
activities.” A few other stakeholders raised tentative concerns about the risk of investing 
time and money in conducting a diligent VV&A effort, only to be denied eventual 
approval because “the bar was set to high.” As was the case with costs, most 
stakeholders downplayed the barriers posed by perceived risks by pointing out that such 
risks were “inherent” in any accreditation effort and would be mitigated by careful 
planning. Thus, most stakeholders viewed the risk of complying to be more of an issue 
of “perception” than a matter of substance. 

5 These are Questions #4, #5, and #6 from the semi-structured interviews. 
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Lastly, the interviews showed that a few of the stakeholders who answered 
Question #6 saw “benefits” in not complying with COMOPTEVFOR 5000.1 A. These 
stakeholders believed that non-compliance would allow PMs to exercise more flexibility 
in their VV&A processes. However, several of the other stakeholders mentioned that 
“tailoring” of the COMOPTEVFOR VV&A process was (or should be) a possibility for 
PMs that would help address the need for flexibility. However, the idea of reaping any 
real benefit from a significant non-compliance with the instruction was a “non-starter” for 
the overwhelming majority of stakeholders interviewed. 

3. Factors for Compliance with COMOPTEVFORINST 5000.1A. 

The factors for compliance with COMOPTEVFORINST 5000.1 A are indicated 
by the stakeholders’ answers to the three interview questions 6 that dealt with the 
“benefits” of compliance, and the “costs” and “risks” of non -compliance. 

In answering interview Question #3, the stakeholders gave answers that point to a 
strong consensus that many significant benefits derived from full PM compliance with 
COMOPTEVFORINST 5000.1 A. The most obvious benefit of compliance cited was the 
high likelihood of Tester/Accreditor (i.e., COMOPTEVFOR) acceptance of the M&S 
proposed for use in addressing a COI under OT. Other benefits observed include the 
higher quality of the M&S being produced and the increased potential to re-use the M&S 
accredited through the process. One stakeholder noted that new participants who joined 
later in the VV&A process seemed better able to see the rationale for the use of the M&S, 
which made for a smoother transition in the program. Several of the stakeholders 
affirmed that the most significant program benefit was the opportunity gained in defining 
problems early, so that they were addressed in time (and within budget). This allowed 
PMs to either make adjustments within the program or take the program in another 
direction without major impact to cost and schedule. Many saw overall program cost 
reduction and standardization of M&S products as significant benefits realized through 
compliance with COMOPTEVFOR’s VV&A instruction. 

6 These are Questions #3, #7, and #8 from the semi-structured interviews. 
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The answers to Questions #7 and #8 indicate that the costs and risks entailed by 
non-compliance make strong dissuaders for any PM deviations from the instruction. 

The vast majority of stakeholders interviewed considered the costs of not 
complying with the COMOPTEVFOR VV&A instruction to be (at least) “several” and 
“significant.” In their comments, the stakeholders seemed to have no trouble coming up 
with concrete (i.e., real dollar value) examples of the costs incurred by non-compliance. 
A major concern was the cost fallout of the M&S not meeting COMOPTEVFOR’s 
technical requirements: Useful M&S would not be produced at the end of all the PM’s 
expenditures because COMOPTEVFOR found insufficient reason to be confident in the 
credibility of the M&S. This adverse outcome could bring an acquisition program to a 
halt, where the cost to go back and re-conduct the VV&A will be more expensive to the 
U.S. Navy. 

Similarly, an overwhelming majority of the stakeholders viewed the risks of not 
complying with the COMOPTEVFOR VV&A instruction to be (at least) “several” and 
“significant.” A number of “real life” examples were provided, which demonstrated the 
seriousness of the risks that non-compliant M&S proponents would take on. 
Stakeholders noted that risks realized are often “felt” in the form of added costs and 
extended schedules. A technical risk in observed for non-compliance was the creation of 
“false positives” with regard to M&S credibility. If the PMs were to serve as their own 
accreditors in the VV&A process, a stakeholder comment posited, “they would be less 
stringent in meeting the M&S use requirements and would run the risk of qualifying an 
M&S that would be deficient (without realizing it).” The results of these false positives 
would be rolled-up into the overall system to which the M&S is being applied, thus 
embedding errors that could lead to bad decisions on the operation of the overall system. 
The unfortunate result is that the warfighter could pay a high price for those 
miscalculations, in some cases with his or her life. 

The consensus demonstrated by the stakeholders on the issues of cost and risk 
may be a good indication of the effectiveness of the COMOPTEVFOR instruction in 
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educating the U.S. Navy surface ship acquisition participants on the importance of 
following the standards of the VV&A process. 

4. Possible Improvements to COMOPTEVFORINST 5000.1A 

Stakeholder interview Question #10 (with insight provided by Question #9) 

provides the data for whether improvements are possible for COMOPTEVFOR 5000.1 A. 

The stakeholders’ comment-answers indicated that no consensus existed regarding the 

opportunity for making improvements; rather, they indicated that there were two 

opposing camps on the issue (and possibly a third one somewhere in the middle). The 

two opposing perspectives were: (a) There is no real opportunity (or impetus) for 

improvement; and (b) There is opportunity for making significant improvements to the 

latest instruction. The first camp was occupied by a minority (6 of 14) stakeholders; the 

second was larger and more productive with comments. The below comment from a 

Tester/Accreditor pointed to an improvement that resonated among the stakeholders: 

Since OT is the final step before full production, it would be most 
efficient to offer the PM a VV&A instruction that contains both 
policy and detailed “how to” guidance a single, usable resource. 

The PM would there find efficient methods to organize, estimate 
costs, determine schedules, and finish a quality M&S on time and 
within budget. 

Thus, many of the stakeholders interviewed appeared to be saying that 
COMOPTEVFORINST 5000.1 A was not sufficient unto itself for providing M&S 
proponents with the information needed to conduct the required VV&A. Acknowledging 
this insufficiency, several of the stakeholders offered specific remedies. A few 
respondents indicated that a “how to” guidance should be included within the instruction 
that would give PMs the ability to establish a more efficient method for organizing, 
estimating costs, and determining schedule impacts. This would assist the PMs in 
producing a quality M&S on time and within budget. Another respondent suggested that 
an appendix should be added to the instruction that would provide detailed examples of 
how to conduct VV&A. This appendix would give a new PM, step-by-step instructions 
and help him/her to better understand the VV&A process. One other recommendation 
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was the implementation of an on-line “distributed knowledge-base” capability for the 
COMOPTEVFORINST 5000.1 A VV&A process. The respondent felt that this would 
not only be innovative to PMs, but would also save money for the VV&A community by 
reducing travel cost to remote sites to attend meetings. 

C. RESEARCH QUESTIONS 

1. Primary Research Question 

The primary research question posed by this thesis is: What are the barriers to 
compliance with COMOPTEVFOR's VV&A process that U.S. Navy surface ship (and 
weapons) acquisition programs face? 

A basic lack of understanding of COMOPTEVFOR's VV&A process and costs 
are the primary barriers stakeholders are facing with not complying with this instruction. 
Some PMs are not aware or lack the understanding that the COMOPTEVFORINST 
5000. lA's process calls for COMOPTEVFOR to get involve early in their program's 
acquisition cycle to determine which assets and resource are appropriate to support 
operational testing, along with identifying the proper modeling and simulation 
requirements which would include operational, programmatic, and perfonnance data. 
Cost savings to programs would be realized if COMOPTEVFOR is involved early, to 
identify or resolve any issues that could impact OT. 

Another barrier is "cost" if not adequately budgeted by PMs for M&S 
development and to conduct V&V activities. "Upfront" costs are necessary for the 
development of credible M&S which is only obtained by developing a well-defined 
VV&A process for the M&S use. Investing early in M&S and VV&A efforts will result 
in a cost savings to the overall program. 

2. First Subsidiary Question 

First subsidiary question: What benefits and liabilities are seen by the 
stakeholders in subscribing to the COMOPTEVFORINST 5000.1 A VV&A process, if its 
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standards were consistently followed by all U.S. Navy surface ship (and weapons) 
acquisition programs? 


a. Benefits seen by Stakeholders: 

Respondents said that PMs and everyone else will gain a common 
understanding of the requirements when they consistently follow the process standards of 
COMOPTEVORINST 5000.1 A. This common understanding will translate to more 
efficient management of the entire acquisition process that will endure. The sum result is 
both a larger dollar savings due to management efficiencies and a higher quality product 
delivered in a shorter period of time. In addition, compliance with the instruction would 
help avoid delays of Defense Acquisition Program Milestone B and the need to 
reprogram funds for physical testing. 

b. Liabilities seen by Stakeholders: 

One respondent perceived that if COMOPTEVFORINST 5000.1 A was 
consistently followed, PMs would not have the opportunity to research new 
methodologies that could be more beneficial and cost-effective in supporting their OT 
requirement. Another respondent felt that PMs might see the accreditation process as too 
much work and therefore would have no reason to reduce testing. This mindset would 
provide leverage for the OTD to require more testing to make up for the lack of support 
that accredited M&S provides. This dynamic saves money up front but the PM will pay 
more later for the additional testing. This is because the PM will learn of problems later 
when they will cost more to fix. Even worse, fewer requirements will be met because 
some of the problems, when realized for the first time at OT, are discovered too late to 
correct. 


3. Second Subsidiary Question 

Second subsidiary question: What are the cost and schedule considerations that 
impact U.S. Navy surface ship (and weapons) acquisition programs in the course of 
complying with the COMOPTEVFORINST 5000.1A VV&A process? 
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There are additional, upfront costs associated with M&S development and VV&A 
process support; however, there are no real added costs associated with adherence to the 
COMOPTEVFOR’s VV&A instruction. In fact, most of the stakeholders interviewed 
argued that the life-cycle cost to use M&S for OT is reduced compared to what would be 
needed to perform physical testing to prove COIs. While some PMs may view VV&A as 
too formal, too cumbersome, and even too costly and time consuming for acquisition 
programs controlled by cost and schedule constraints, this view, as articulated by a 
stakeholder, appears to be shortsighted and unwise from the total acquisition cost and 
schedule perspective. 

4. Third Subsidiary Question 

Third subsidiary question: What are the risks observed by the stakeholders when 
the COMOPTEVFORINST 5000.1 A VV&A process is not followed? 

As indicated by the stakeholders, the risks associated with non-compliance of this 
instruction will include missed opportunities to adequately plan the M&S and VV&A 
objectives needed to obtain a continual improvement process throughout the ship’s 
acquisition life-cycle. In addition, opportunities would also be lost in making strategic, 
up-front investments in M&S and VV&A such as design & trade studies, pre-test 
planning, and pre-test predictions; the lack thereof results in poor quality and 
unaccredited M&S. Ultimately, OT test requirements would not be fully satisfied due to 
a lack of credibility in the M&S. 


5. Fourth Subsidiary Question 

Fourth subsidiary question: What actions can be taken by stakeholders to ensure 
compliance with COMOPTEVFOR's VV&A process? 

In order for the four stakeholder groups to ensure compliance with 
COMOPTEVFOR's VV&A process, they must not only do their individual part but also 
work collectively as a group. For instance, the VV&A standards group should not only 
be responsible for the coordination and integration of M&S efforts across all U.S. Navy 
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warfare and support areas, but they should also be actively communicating and 
encouraging the use of VV&A standards to the other stakeholder groups. Likewise, in 
addition to identifying the operational requirements for the particular weapon system and 
providing the required funding to support that system, the resource sponsor should be 
routinely engaged with the PM to ensure that the VV&A requirements are being met and 
appropriate funding is allocated in that area. Although the PM is ultimately responsible 
for selecting or developing the M&S and ensuring that it is accredited to support its 
intended use, he/she should make sure that COMOPTEVFOR (tester/accreditor) is 
involved in the early stages of the M&S development and VV&A process to provide 
operational insight so that any issues or constraints can be addressed and corrected early 
enough to avoid any delays during OT. Finally, the tester/accreditor, whose main 
concern is obtaining an M&S of sufficiently high fidelity as it relates to accurately 
modeling the physical behavior of the system under test, should also be sensitive to the 
cost and schedule constraints of the acquisition program. If these synergies were 
maintained among the stakeholders, they will be ensuring compliance with the 
instruction. 

D. RECOMMENDATION 

The stakeholders, working together, should diligently comply with the VV&A 
process advocated by COMOPTEVFORINST 5000.1 A in developing the M&S needed in 
support of U.S. Navy surface ship OT requirements. However, this can be an issue for 
PMs when the upfront costs to support the requirements specified in the instruction are 
not in their original acquisition program budget. Early interaction between the 
stakeholders, particularly between the PM and the Tester/Accreditor, is crucial to 
addressing this issue. The VV&A Standards stakeholder can provide considerable 
information and support to the PM and the Tester/Accreditor that would help anticipate 
the cost and schedule requirements. Ultimately, the Resource Sponsor can broker the 
expectations between the PM and the Tester/Accreditor and approve the budget. 

A solution that would help absorb the possible VV&A cost deltas above any one 
specific acquisition program RDT&E baseline would be to establish a special RDT&E 
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funding line dedicated to the VV&A of M&S for OT. Appropriated under OPNAV N76 
cognizance, this RDT&E funding would be potentially available to all surface ship 
acquisition programs when the need arises; the line would be budgeted yearly to support 
the PMs if the effort to support the COMOPTEVFORINST 5000.1 A VV&A process isn’t 
in their individual budgets. PMs should have to provide the appropriate justification 
when requesting this funding to support their OT requirements. This special RDT&E 
funding line would also support M&S reuse among surface ship acquisition programs, 
since the N76 Resource Sponsor would likely encourage such efforts. 

Also, to better aid PMs and Resource Sponsors in understanding the benefits and 
advantages of complying with the COMOPTEVFORINST 5000.1 A VV&A process, 
NSMO, in conjunction with COMOPTEVFOR, should provide training/seminars to the 
surface ship PMs and OPNAV on this instruction. 

If these recommendations were to be implemented by the U.S. Navy, the 
researchers believe that there would be a higher incidence of full compliance by PMs in 
utilizing the COMOPTEVFORINST 5000.1 A VV&A process, thus allowing the U.S. 
Navy to achieve the ultimate goal of producing more credible M&S to resolve COIs that 
satisfy COMOPTEVFOR’s OT requirements for surface ships and their weapon systems. 


E. AREAS FOR FURTHER RESEARCH 

The following areas should be investigated for potential benefit to the U.S. Navy, 
the other U.S. military services, and DoD components: 

1. Accrediting Legacy M&S 

One of the findings of this thesis was why a well-defined VV&A process for 
newly acquired or modified surface ships is important in gaining credibility for the M&S. 
Further research in defining the steps that should be taken in collecting or re-producing 
the background data that is needed to validate legacy models could provide further 
insight in the importance of obtaining credible M&S. 
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2. The “How To” of VV&A 

This thesis focused on the organizational and managerial aspects of VV&A of 
M&S used for OT of U.S. Navy surface ships and their weapon systems. Work is being 
done by the U.S. Navy’s VV&A Standards group, namely NMSO, to produce a "how to" 
handbook on the VV&A process. The handbook therein is expected to address a specific 
need identified by a number of the stakeholders interviewed. Further research can be 
done by collecting data on the VV&A process perfonnance of stakeholders before and 
after issuance of the Implementation Handbook to further provide guidance to the 
community on M&S VV&A. 

The areas that are identified for future research will help to increase the 
knowledge-base of both stakeholders and members of the M&S VV&A community in 
identifying new approaches and techniques that would lead to more credible and cost- 
effective M&S for not only the U.S. Navy, but other U.S Military services and DoD 
components as well. 
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APPENDIX: INTERVIEW DATA 


A questionnaire was administered to 14 stakeholders of the U.S. Navy’s VV&A 
of M&S used for surface ship OT&E. It provided the basic structure of the stakeholder 
interviews, by soliciting answers - in the form of a score and a comment - to each of ten 
questions. The interviews were conducted on a non-attribution basis; the stakeholders’ 
names and U.S. Navy commands are not identified in order to protect their 
confidentiality. Instead, a letter designation (e.g., [A] ) is assigned to each stakeholder. 
Further, the stakeholder category (e.g.. Tester / Accreditor {TA} ) and management level 
(e.g., Lower Management {lm} ) is identified for each stakeholder. Table A. 1-0, below, 
lays out the interviewed stakeholder identifications and the counts per category and level. 


Table A.1-0. Interviewed Stakeholder Identifications and Counts 

Stakeholder Category 

Lower 

Management 

{lm} 

Middle 

Management 

{mm} 

Upper 

Management 

{urn} 

Count per 

Category 

Tester / Accreditor {TA} 

[A] [B] 

[C] 

|D] [E] 

5 

Resource Sponsor {RS} 

[H] 

[G] 

[I] 

3 

Program Manager {PM} 

[K] [M] 

[L] [N] 

[O] 

5 

VV&A Standards {VS} 



\P] 

1 

Count per Management Level 

5 

4 

5 

14 


The questions asked and the answers provided in the course of the interviews are 
recounted below. The first table below each question lists the choices of numerical 
scores offered to the stakeholders in answering the question. The scores provided as 
answers are reported in the second table below each question. The stakeholders’ 
comment-answers for each question are detailed in alphabetical order of stakeholder 
identifier below the second table of each question. Question 1 (only) had two sub- 
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questions, which were used to elicit amplifying information from the stakeholders’ about 
their specific experiences in VV&A. 


QUESTION 1. 

To what extent are you familiar with the VV&A process required by the newly updated 
COMOPTEVFORINST 5000.1 A of 9 September 2004, as it applies to USN surface ship 
M&S used for OT&E? 


Table A.l-1. Choices of Score Answers Offered to Stakeholders for Question 1. 


(5) 

(4) 

(3) 

(2) 

(1) 

very 

moderately 

somewhat 

moderately 

very 

familiar 

familiar 

familiar 

unfamiliar 

unfamiliar 


Table A.l-2. Scores Provided by Stakeholders to Question # 1 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

5 

5 

5 

1 

2 

2 

4 

4 

4 

2 

5 

4 


Sub-Questions for Question 1 : 

a. What is one particular case of VV&A of USN surface ship M&S used for 
OT&E in which you participated and what was your role in this case? 

b. Was the superseded COMOPTEVFORINST 5000.1 (of 5 September 
1995) in effect for this case? 

Comments / Answers Provided by Stakeholders for Question 1: 

Stakeholder fAl {TA, Im} : 

• I am very familiar (at the higher level) with the instruction. 

• I participated in VV&A for Rolling Airframe Missile (RAM). I supported 
the OTD in developing OT M&S requirements and evaluating V&V results. I also 
participated in VV&A for Missile Defense (which had equipment test failures) and for 
Tomahawk (all sections, using five simulators), which enabled minimal physical testing, 
but required (all together) a long, detailed process. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for the VV&A efforts in which I participated. I supplemented the 
instruction with our Surface Warfare Guide. (This is what the PMs asked for and needed 
to accomplish their VV&A responsibilities.) 
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Stakeholder FBI {TA, Im} : 

• I am very familiar with the instruction. 

• The one particular case of VV&A in which I participated was Tomahawk. 
My role in this case was that of Operational Test Director (OTD), where I attended SCPs 
(vs. SMBs) and was involved in the review of five PM-produced VV&A documents: (1) 
Accreditation Plan; (2) Accreditation Report; (3) Accreditation Test Report; (4) 
Configuration Management Plan; and (5) Overarching VV&A Plan. I helped the PMs 
with their VV&A by providing the “Guide for U.S. Navy Surface Ship VV&A of M&S 
for OT&E.” 

• The key to successful M&S is user involvement in the VV&A. The user 
of the weapons system is the Fleet and COTF represents the user in testing and 
accrediting their system. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for this case. The last cycle of Tomahawk testing / M&S was conducted 
under the old instruction. 

Stakeholder [Cl 1TA, mm}; 

• I am very familiar with the instruction. 

• I participated in the VV&A for one ECS (early stage) case, two CVN-21 
cases, one LPD-17 case, and a number of others that are related to USN surface ship 
programs. My role in these cases was that of COMOPTEVFOR’s VV&A manager, 
where I reviewed and approved the VV&A of M&S for OT&E documents submitted by 
the USN Surface Ship Program Managers. I am responsible for the migration of the 
COMOPTEVFOR instruction that you are studying (COMOPTEVFORINST 5000.1 A of 
9 September 2004). 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for most of these cases, but since I wrote the new instruction, it was in effect 
for the most recent case. 

Stakeholder FD1 {TA, um}: 

• I am very familiar with the instruction. 

• The one particular case of VV&A in which I participated was reviewing 
new policy as it went through the approval / signature process. My role in this case was 
that of newly appointed, interim VV&A Manager at COMOPTEVFOR. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for this case. I was a part of approving the new instruction. 
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Stakeholder [El {TA, um}: 

• I am very familiar with the instruction. 

• The one particular case of VV&A in which I participated was reviewing 
(at the senior management level) new policy as it went through the approval / signature 
process. My role in this case was that of developing the long-term strategic policy with 
respect to the U.S. Navy and OT&E. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for this case. I was a part of approving the new instruction. 

Stakeholder FG1 IRS, mm}: 

• I am very unfamiliar with the instruction. 

• I oversaw the DD(X) TEMP process from adjudication to signature. This 
included general oversight of VV&A conducted in support. My responsibilities include: 

— Defining requirements for Surface Navy (the DD(X) program, in 
particular). 

— Providing requirements to the PM for product by way of the 
Capabilities Design Document (CDD). 

— Reviewing/authorizing the PM’s plan on how the product will be 
implemented. 

— Providing funding to the PM to implement the product. 

— Maintaining balance between PM & COTF (and DOT&E) with respect 
to program costs & risks. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for this case. 

Stakeholder [HI IRS, Iml: 

• I am moderately unfamiliar with the instruction. 

• The one particular case of VV&A in which I participated was where two 
systems for Tomahawk were evaluated: the TTWCS upgrade ver 5 and the TC2S ver 4.1. 
My role in this case was that of Resource Sponsor for Tomahawk Weapons System. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for this case. 

Stakeholder [II IRS, um}: 

• I am moderately unfamiliar with the instruction. 

• There are a number of cases of VV&A in which I participated: 
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— USS Virginia (SSN 774) Program VV&A of a Transient Shock 
Analysis (TSA) “Process.” (This was recently completed and allowed 
elimination of a $70 Million dollar Full Ship Shock Trial.) 

-- DD(X) ASCM PRA Test Bed. (This is ongoing.) 

— Joint Lightweight Standoff Chemical Agent Detector (JLSCAD). 

— Netcentric Warfare Knowledge Sharing. 

- LPD-17, LHA-6. and CVN-21 

• My role in these cases was that of the U.S. Navy’s T&E oversight, where I 
review the VV&A to see that the M&S is used appropriately for T&E (whether for 
DT&E or OT&E). While COTF is a tester, OPNAV N091 is not (though we were). 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for the older cases; but was not was not in effect for the most recent cases. 

Stakeholder [K1 1PM, Im}: 

• I am moderately familiar with the instruction. 

• The one particular case of VV&A in which I am participating is the 
DD(X) program, though it is not at that stage yet. However, we are planning for the 
VV&A to the extent possible. My role in this case was that of project engineer. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) is 
not in effect for this case. Thus, we are using the new instruction. 

Stakeholder [LI 1PM, mm}: 

• I am moderately familiar with the instruction. 

• The one particular case of VV&A in which I participated was the Threat D 
M&S Suite to support TEMP 124/801 OT-IIG. My role in this case was that of 
managing the VV&A of the Threat D M&S Suite to support TEMP 124/801 OT-IIG. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for this case. 

Stakeholder [Ml 1PM, Im}: 

• I am moderately familiar with the instruction. 

• The one particular case of VV&A in which I participated was for the 
LPD-17 Probability of Raid Annihilation (PRA) V&V. My role in this case was that of 
program analyst. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for this case. 
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Stakeholder [N1 {PM, mm}: 

• I am moderately unfamiliar with the instruction. 

• The one particular case of VV&A in which I participated was LPD-17 
Probability of Raid Annihilation (PRA) for ship self defense. My role in this case was 
that of supporting PM in the development of the M&S and contributing to the VV&A 
case. 


• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for this case. 

Stakeholder FOl {PM, um}: 

• I am very familiar with the instruction. 

• The one particular case of VV&A in which I participated was 
TechEval/OpEval for CEC; OT for ESSM; OT for SPYID(V); and ABMD for COTF. 
My role in this case was that of overseeing the VV&A processes and assisting the PMs in 
performing VV&A. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was in effect for these cases. 

Stakeholder [PI (VS, um}; 

• I am moderately familiar with the instruction. 

• I participated in all of the recent VV&A cases, as I provide the Navy’s 
oversight on all VV&A and communicate / coordinate the Navy’s standards on VV&A. 

• The superseded COMOPTEVFORINST 5000.1 (of 5 September 1995) 
was not in effect for the most recent cases. 

QUESTION 2. 

What level of compliance with the (effective) instruction do you think was achieved in 
this case? 

Comments Provided by Stakeholders for Question 2: 

Stakeholder fAl {TA, Im} ; 

• I think that 100%-91 % compliance was achieved. 

• A higher level of compliance was achieved for those PMs who used 
COTF’s supplemental Surface Warfare VV&A Guide. 

• The process diagram developed by NDAT was helpful in establishing a 
standard process and schedule. 
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Table A.2-1. Choices of Score Answers Offered to Stakeholders for Question 2. 


(5) 

(4) 

(3) 

(2) 

(1) 

100-91% 

90-81% 

80-71% 

70-61% 

60% or less 

compliance 

compliance 

compliance 

compliance 

compliance 


Table A.2-2. Scores Provided by Stakeholders to Question # 2 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

4 

1 

1 

1 

5 

5 

4 

5 

4 

4 

5 

5 


Stakeholder FBI {TA, lm} : 

• I think that 100%-91 % compliance was achieved. 

• The VV&A efforts of one particularly successful program consistently 
demonstrated a high level of compliance with respect to “content”, though not as high 
with respect to “schedule.” The real objective for compliance is to have “no surprises” 
with respect to the content of the VV&A produced for the M&S. 

• Successful compliance was achieved when a COTF representative 
participated on the SCP. 

• Both PMs and COTF were usually not successful in meeting VV&A 
timelines. Then again, failing to meet the original schedule is usually not the real 
problem. 

Stakeholder [Cl {TA, mm} : 

• I think that 90%-81 % compliance was achieved. 

• As the new instruction has hardly been in place, no one has yet had the 
opportunity to demonstrate a level of compliance for it. 

Stakeholder FD1 IT A, um}: 

• I think that 60% or less compliance was achieved. 

• Meeting VV&A schedules is an issue. 

• In one case, the PM did not provide the data to finish the VV&A package. 
So the data had to be developed by the OT himself. 

• The old instruction (which applied) was not very effective. The new 
instruction will be more effective. 
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Stakeholder [El {TA, um}: 

• I think that 60% or less compliance was achieved. 

• A very low level of compliance was achieved. Most PMs did not seem to 
be knowledgeable about the instruction. Timing (i.e., late delivery) was the usual cause 
of non-compliance. 

Stakeholder FG1 IRS, mm}: 

• I think that 60% or less compliance was achieved. 

• A case of low compliance was recognized when the Resource Sponsor 
(RS) ran into a big issue at the end of the VV&A process: DOT&E was not satisfied with 
the capability of the M&S to credibly predict the perfonnance of the weapon system 
modeled. The RS wanted to use M&S to avoid the cost and risk entailed in conducting a 
certain close-in live-fire testing event. But the PM had not engaged the TA sufficiently 
(or early enough) in their VV&A process to identify the shortcomings of the M&S, such 
that DOT&E rejected the use of the M&S for the close-in event. Consequently, the RS 
had to reprogram $250M to conduct a physical test that the M&S was hoping to avoid. 

Stakeholder [HI IRS, Im}: 

• I think that 100%-91 % compliance was achieved. 

• The maturity and experience of the team that develops the ship’s T&E 
program, which includes M&S in its mix, has a lot to do with any successful compliance 
with the VV&A instruction. 

Stakeholder [II IRS, um}: 

• I think that 100%-91 % compliance was achieved. 

• One recently completed VV&A effort enabled the elimination of a $70M 
physical T&E event required for a major ship acquisition. This success was achieved 
because the PM employed a “test-model-test” iterative effort over a number of years in 
the ship’s RDT&E program to collectively understand the physics of physical event on 
the ship. Further, the ship’s designs, tests, and models were planned to work in support 
of each other. The ship’s subassembly construction techniques facilitated M&S and 
VV&A. Finally, M&S process control was achieved to at least CMMI level 3 standards. 
This facilitated a rapid VV&A process. 

Stakeholder [K1 1PM, Im}: 

• I think that at least 90-81% compliance will be achieved. 

• The PM is trying to achieve a high level of compliance (at least 90% 
compliance) and - among other efforts - attended VV&A TWG meetings to learn how to 
do so. 
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Stakeholder [LI {PM, mm}: 

• I think that 100%-91 % compliance was achieved. 

• The PM worked very closely with the OTD to ensure that the PM’s 
process was in accordance with COTF policy and met their requirements. 

Stakeholder [Ml 1PM, Im}: 

• I think that at least 90-81% compliance will be achieved. 

• The COTF instruction is fairly general. The PM believes that his VV&A 
program will meet the instruction’s intent. COTF was engaged early on in providing 
input to the process. However, the instruction was not written with the scale of the 
program’s M&S in mind. 

Stakeholder [N1 1PM, mm}: 

• I think that at least 90-81% compliance will be achieved. 

• The VV&A case on which the PM is working is still under development. 
The PM thinks that he will be highly compliant. 


Stakeholder FOl 1PM, uml: 

• I think that 100%-91 % compliance was achieved. 

• The PM believes that he achieved a high level of compliance for the 
VV&A efforts in which he participated. 

Stakeholder [PI {VS, um}: 

• I think that 100%-91 % compliance was achieved. 

• It appears that a high level of compliance is being achieved, but it can be 
done more consistently and efficiently. 

QUESTION 3. 

What do you see to be the benefits associated with compliance with this instruction? 
Comments Provided by Stakeholders for Question 3: 

Stakeholder fAl {TA, lm} : 

• The benefits of complying with the COTF instruction are augmented when 
PMs use the M&S Surface Warfare Guide supplemented by the COTF OTDs. The guide 
helps PMs (and everyone else) better understand what is required (especially any OT 
related clarifications / details). The combined use of the instruction and the guide enables 
more efficient management of the entire process over the years. Consequently, there are 
significant financial savings due to the management efficiencies. Also, a higher quality 
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product is delivered in a shorter period of time, which again provides a financial savings 
(that is obvious but hard to quantify). 


Table A.3-1. Choices of Score Answers Offered to Stakeholders for Question 3. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 

minor 

real 

benefits 

benefits 

benefit 

benefits 

benefit 


Table A.3-2. Scores Provided by Stakeholders to Question # 3 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

5 

4 

5 

4 

4 

5 

3 

5 

4 

5 

5 

4 


Stakeholder FBI {TA, Im{ ; 

• Benefits seen from compliance include: 

— A higher quality of M&S product is obtained. 

— A better assurance that M&S will meet the needs of the user. 

— OTDs are satisfied that OT requirements are being met. 

Stakeholder [Cl {TA, mm} : 

• Re-use of the M&S is one of the many benefits. When the PMs comply 
with the effective VV&A process, COTF will eventually certify the use of the M&S. The 
accreditation process is built in. When the program complies with the instruction, any 
new participants who join later in the process can readily see the rationale for the use of 
the M&S. 

Stakeholder [D1 {TA, urn} : 

• Defining the intended use of the model is the most important benefit. 
Compliance with the instruction gives the proponent a mechanism to support the 
appropriate use of the M&S and bring the process to a recognized conclusion. 

Stakeholder [El {TA, urn) : 

• Compliance leads to obtaining more validated data that other proponents 
can use (i.e., re-use). 

• The bottom-line is that complying with the instruction successfully obtains 
the accreditor’s acceptance of the use of the M&S. 
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Stakeholder TGI {RS, mm} : 

• Compliance with the instruction helps PMs avoid: 

— DODI 5000.2 [particularly sections E5.1 and E5.10]. 

— Delays in reaching Defense Acquisition Milestone B. 

— The need to reprogram RDT&E funds to support physical testing 
(vs. M&S). 

Stakeholder [HI {RS, lmj : 

• Compliance helps promote standardization and commonality. When you 
standardize, you save costs (and cost reduction is one benefit). 

• One particular Navy weapons program made heavy use of M&S for tens 
of years. What is in place today has saved a lot of money. The program has end-to-end 
testing and M&S capability, which provides flexibility in using different modes, and uses 
an iterative model update process. The program is able to work a combination of 
modeling and simulation, where the simulation generates environmental inputs into real 
hardware so that the weapon’s mission can be run in a very realistic way. 

• One of the significant program benefits of full compliance is that problems 
are defined early, so that they can be addressed in time and within budget. 

Stakeholder [II IRS, urn}: 


• Compliance with the COTF instruction supports compliance with several 
other DoD and DoN regulations for Defense Systems Acquisition that address T&E and 
M&S, including: 

— DODI 5000.2 [particularly sections E5.1 and E5.10]. 

-- SECNAVINST 5000.2C [particularly section 5.4.7.9]. 

- SECNAVINST 5200.40. 

Stakeholder [K1 {PM, lm} ; 

• Compliance entails earlier involvement in the definition of high-level 
M&S requirements and their associated acceptance criteria. This obtains considerable 
time and cost savings in the long run. 

Stakeholder [LI 1PM, mm} : 

• COTF required a high degree of rigor in their VV&A process (as called 
out in their instruction) and were supportive and helpful to the PM in achieving that rigor. 
As a consequence, they “forced” the PM to be sure that his data collection methodology, 
analysis processes, and reporting procedures were sufficient to meet the demands of the 
Accreditation Review Panel (ARP). Consequently, the PM is 100% certain that his 
VV&A would stand up to outside audit. 
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Stakeholder TM1 {PM, lm} : 

• By complying with the instruction, the PM interacted with COTF, the 
accreditation authority, and benefited from their experience and guidance. 

Stakeholder INI |PM, mm} : 

• Compliance with the instruction promotes standardization and reuse, 
where it would otherwise only be done at a low percentage. This benefits more than the 
individual program and sets the groundwork for others to build and reuse. 

Stakeholder jOl {PM, urn} : 

• There are many benefits to compliance, including that it allows re-use of 
M&S and it promotes proper documentation. 

Stakeholder [PI {VS, urn} : 

• The benefits of compliance come under the categories of (1) Cost- 
reduction and (2) Standardization. 


QUESTION 4. 

What do you see to be the costs associated with compliance with this instruction? 


Table A.4-1. Choices of Score Answers Offered to Stakeholders for Question 4. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 

minor 

real 

costs 

costs 

cost 

costs 

cost 


Table A.4-2. Scores Provided by Stakeholders to Question # 4 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

4 

4 

1 

2 

2 

2 

2 

1 

2 

3 

1 

4 

4 

1 


Comments Provided by Stakeholders for Question 4: 

Stakeholder [A] {TA, lm} : 

• I see several significant costs associated with compliance. 

• When it comes to compliance, there is no “free lunch.” In conducting 
VV&A, due diligence (in planning, organizing, executing, and documentation) costs 
money. 
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Stakeholder FBI : 

• I see several significant costs associated with compliance. 

• Costs are incurred in both the financial and schedule realms, and are 
particularly evident: 

— During development of the M&S. 

— When performing the V&V. 

— During collection and review of data. 

— At feedback to stakeholders. 

— With the group doing the M&S when it is not meeting the overall 
schedule. 

• New models require paradigm shifts (incurring costs and schedule 
increases). Legacy models are more readily accepted and take less time to be accepted. 

Stakeholder [Cl : 

• I see no real cost associated with compliance. 

• PMs see significant costs from compliance in having to expend ‘political 
capital’ and having to pay more for “internal education processes.” 

• There are costs in preparing the VV&A documentation. Compliance 
requires a high level of interaction between COTF and the PM (entailing planning costs) 
that achieves early, up-front recognition of the entire life-cycle cost of the M&S. This is 
needed, as it should avoid the cost surprises often discovered down the OT&E road. 

• Ultimately, there are no genuine added costs associated with compliance. 
It could be argued, further, that the total cost to use M&S for OT&E is probably less than 
that which would be needed to perform physical testing. 

Stakeholder FD1 : 

• I see a few minor costs associated with compliance. 

• Any costs to comply are minor because the PM should otherwise be doing 
(e.g., for his program’s DT&E) what the COTF instruction requires for OT&E. Thus, the 
instruction provides a framework for doing VV&A correctly. Compliance with the 
COTF instruction supports a methodology that is consistent with all acceptable W&A 
efforts. 

Stakeholder [El : 

• I see a few minor costs associated with compliance. 

• The PM would end up paying about the same costs, regardless of whether 
they are trying to comply with the COTF instruction, in any genuine effort to prove that 
their models are credible for the intended use. 
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• The question to be answered when deciding whether or not to use an M&S 
for OT&E is: “What money would be saved in OT&E if this M&S were to be used 
instead of a physical test?” 

Stakeholder [G1 : 

• I see a few minor costs associated with compliance. 

• Compliance with the instruction means more “up-front time” and man¬ 
hours devoted to PM - COTF interactions than would the typical PM (in all likelihood) 
devote to VV&A conducted without the obligation of the instruction. However, without 
the instruction, the PM would end up having to collect more ‘real’ data (thereby incurring 
more cost) to make the M&S credible. 

Stakeholder [HI : 

• I see a few minor costs associated with compliance. 

• Compliance may bring about higher up-front costs, but down-stream costs 
(and consequently total costs) are lower. 

• Mature (seasoned) acquisition programs meet routinely with COTF. 
Consequently, their program costs for VV&A are lower as these meetings are simply a 
part of their program. But for new programs, the VV&A meeting costs would likely be 
higher. 

Stakeholder [II : 

• I see no real cost associated with compliance. 

• The cost of compliance is not the issue. Rather, the focus should be on 
meeting the technical challenges. 

• While some PMs may view VV&A as too fonnal, too cumbersome, too 
costly and time consuming to a program controlled by cost and schedule constraints, this 
view is shortsighted and not true from the total acquisition cost and schedule perspective. 

Stakeholder ]_K[ : 

• I see a few minor costs associated with compliance. 

• The stringent documentation and configuration management requirements 
demanded by the instruction will most likely increase costs for the developers. 

Stakeholder [LI : 

• I see significant cost associated with compliance. 

• Doing VV&A right took time. From start to finish it took over thirteen 
months to move a very complex M&S through the hurdles. In this case, a mix of 
government and contractor personnel already under task was used to perform virtually all 
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phases of the VV&A. Hiring outside (new) support would have significantly increased 
the financial cost. 

• It appears that some DoN field activities are making VV&A unnecessarily 
complex in order to create their own cottage industry of VV&A experts. (This creates 
unnecessary costs.) 

Stakeholder [Ml ; 

• I see no real cost associated with compliance. 

• The instruction is so general that any good engineering practice would 
result in compliance. (Thus, compliance entails no additional costs.) 

Stakeholder FN1 : 

• I see several significant costs associated with compliance. 

• There were significant costs for one program, but costs further down the 
line will be reduced because of the introduction of an enterprise solution for multiple 
programs. The enterprise VV&A plan will enable this to become a cost savings reality. 

Stakeholder FOl : 

• I see several significant costs associated with compliance. 

• The costs associated with compliance depend on risk and the limitations of 
the M&S. 

Stakeholder [PI : 

• I see no real cost associated with compliance. 

• Compliance with the instruction really provides a savings to the DoN. 

QUESTION 5. 

What do you see to be the risks associated with compliance with this instruction? 


Table A.5-1. Choices of Score Answers Offered to Stakeholders for Question 5. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 

minor 

real 

risks 

risks 

risk 

risks 

risk 


Table A.5-2. Scores Provided by Stakeholders to Question # 5 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

1 

2 

1 

2 

2 

2 

1 

3 

1 

1 

2 

2 

1 
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Comments Provided by Stakeholders for Question 5: 

Stakeholder [A1 {TA,Im} : 

• I see one significant risk associated with compliance. 

• One significant risk in complying with the instruction is that PMs 
sometimes see M&S accreditation as a reason to reduce future testing. With less testing, 
there is less new data to plug into future M&S. Consequently, the programmers fall into 
the trap of “fixing” the next M&S to correlate with the previous, limited (and possibly 
less relevant) test data set. The margin of error between the results of a test and the 
output of its M&S is likely to widen, meaning that the M&S has become of lower quality. 
This can lead to significantly higher downstream costs due to all the problems associated 
with poor quality. 

Stakeholder FBI : 

• I see no real risk associated with compliance. 

Stakeholder [Cl : 

• I see a few minor risks associated with compliance. 

• There are political risks, which are inherent in the business of VV&A. 
Compliance with the instruction makes some of the big risks go away, but at the same 
time introduces a few new, smaller ones. 

Stakeholder FD1 : 

• I see no real risk associated with compliance. 

• The greatest risk the PM could encounter by dutifully complying with the 
instruction would be to discover, through the M&S for OT&E process, that the system to 
be acquired does not deliver the required capability and consequently should not be 
fielded. But this is precisely what OT&E is intended to discover and thus an inherent risk 
(whether M&S or physical testing is used) for the PM who presents his system for 
OT&E. 

Stakeholder [El : 

• I see no real risk associated with compliance. 

Stakeholder [G1 : 

• I see a few minor risks associated with compliance. 

• Early interaction with the Tester / Accreditor could require more work by 
the PM than he planned and could drive some of his M&S costs. 

• Resource sponsors do not engage at the level of the instruction to 
determine compliance and non-compliance of VV&A requirements. Rather, they seek 
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early interaction with the PM and the Tester / Accreditor to determine the front lines of 
risk and cost. If the M&S issues do not get figured out by Milestone B, then the Tester / 
Accreditor can stop the process. The risk from non-compliant VV&A is that the use of 
M&S could be disapproved late in the program, which would cause significant cost 
increases and schedule delays. 

Stakeholder [HI : 

• I see a few minor risks associated with compliance. 

• With mature programs, there are no major risks. But at the PM or ‘trench’ 
level, there can be concerns about the programmatic impacts of the ‘outside’ evaluation 
required by the instruction. 

Stakeholder [II : 

• I see no real risk associated with compliance. 

• As with the cost question, the risk of compliance is not the issue. Rather, 
the focus should be on the technical challenges. 

• The PM often views M&S (and consequently its VV&A) as merely a tool 
to meet a requirement; rather, it is an integral process within acquisition that reduces 
program risk. 

Stakeholder [Kl : 

• I see one significant risk associated with compliance. 

• There is the risk of last minute changes to test requirements or objectives, 
which may cause delays in accreditation (and thus test schedule) because stepping back 
the process will require rework on many artifacts and add extra V&V activities. 

Stakeholder [LI : 

• I see no real risk associated with compliance. 

• There is more risk in not complying with the COTF direction than there is 
in the act of compliance. If the PM doesn’t comply, then his VV&A, and hence his 
M&S, will not be valid to use for OT. What could be more risky than developing M&S 
that has no use in OT? 

Stakeholder [Ml : 

• I see no real risk associated with compliance. 

• Meeting the intent of the instruction is concomitant with good engineering 
practice. (Thus, risk is appropriately managed.) 
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Stakeholder [Nl : 

• I see a few minor risks associated with compliance. 

• VV&A planning should go hand in hand with engineering software 
development. No significant risk should arise with compliance. However, there are 
minor risks that come from the instruction’s effect of driving PMs to comply to processes 
that produce relatively ‘safe’ (less pioneering) outcomes rather than encouraging these 
PMs to tackle difficult technical issues that, while they may not lead to as high a 
percentage of compliance, still produce greater M&S capability. 

Stakeholder FOl : 

• I see a few minor risks associated with compliance. 

• If PMs comply, then the risks for their VV&A are very limited. 

Stakeholder [PI : 

• I see no real risk associated with compliance. 

• There is no real risk associated with compliance because all OT&E 
programs should be in compliance. 


QUESTION 6. 

What do you see to be the benefits associated with non-compliance with this instruction? 


Table A.6-1. Choices of Score Answers Offered to Stakeholders for Question 6. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 
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benefits 

benefit 

benefits 

benefit 


Table A.6-2. Scores Provided by Stakeholders to Question # 6 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

1 

1 

1 

1 
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1 

1 

1 

1 

2 

1 

1 

1 


Comments Provided by Stakeholders for Question 6: 

Stakeholder fAl 1TA, Im} : 

• I see one significant benefit associated with non-compliance. 

• From the OT perspective, there may be (perversely) one significant benefit 
associated with non-compliance. In the event the PM has not complied, he has thus not 
justified his proposal to reduce the physical testing required for OT. This provides 
leverage for the OTD to require more testing to make up for the lack of support that 


134 





accredited M&S provides. So, while non-compliance may save money on M&S at the 
front end, the PM ends up paying more for testing at the back end. Moreover, physical 
testing uncovers problems later when they cost more to fix (if they can be fixed at that 
point), which risks meeting fewer requirements. 

Stakeholder [B1 ; 

• I see no real benefit associated with non-compliance. 

Stakeholder [Cl : 

• I see no real benefit associated with non-compliance. 

• Non-compliance produces no real benefit. Rather, the VV&A participants 
should have a fonnal understanding of the model’s intended use. They should identify 
the ‘use cases’. It is important to understand the implications of using one type of model 
over another. The behavior of the model should be consistent with our understanding of 
the physical phenomena. The Configuration Manager has to be clear on intended use. 

Stakeholder FD1 : 

• I see no real benefit associated with non-compliance. 

Stakeholder [El : 

• I see no real benefit associated with non-compliance. 

Stakeholder FG1 : 

• I see no real benefit associated with non-compliance. 

• There is no real benefit that would come from avoiding early interaction 
with COTF to help make the M&S credible. 

Stakeholder [HI : 

• I see no real benefit associated with non-compliance. 

• In practice, there isn’t any real benefit from non-compliance. 

Stakeholder [II : 

• I see no real benefit associated with non-compliance. 

• This question doesn’t really make sense, because non-compliance with 
instruction brings no benefit whatsoever. 

• Unfortunately, PMs at the start of their programs often don’t know what 
they want to use M&S for. Full compliance is difficult when the need for M&S is 
recognized only later. Then, the results of the M&S come too late to benefit the overall 
system design. There are many other VV&A issues commonly associated with non- 
compliance: 
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— M&S activities aren’t coordinated with acquisition strategy. 

— T&E strategy doesn’t provide validation data. 

— Models aren’t accredited prior to use. 

— M&S isn’t planned throughout acquisition lifecycle. 

— M&S use and VV&A strategy aren’t in TEMP. 

— Models are seldom V&V’ed - and if done, seldom prior to use. 

— M&S and VV&A end-states are often not agreed upon at program 
start, resulting in lack of commitment to resource them. 

— Proprietary models cannot be VV&A’ed without contract support. 


Stakeholder [Kl : 

• I see no real benefit associated with non-compliance. 

• Not complying with COTF instruction would cause much more pain than 
benefit (if any). However, PMs should find ways to tailor the instruction to fit within the 
constraints of their acquisition programs without having a negative effect on the integrity 
of the VV&A process. 

Stakeholder [LI ; 

• I see no real benefit associated with non-compliance. 

• There is no real benefit associated with non-compliance whatsoever. 

Stakeholder [Ml : 

• I see a few minor benefits associated with non-compliance. 

• A few minor benefits may be associated with non-compliance. The PM 
may use a different (nevertheless valid) methodology. And sometimes it may be more 
workable to complete M&S accreditation after the model’s use. 

Stakeholder FN1 : 

• I see no real benefit associated with non-compliance. 

Stakeholder FOl : 

• I see no real benefit associated with non-compliance. 

• Not complying with the instruction means that proponents will not be 
conducting a genuine VV&A process. 

Stakeholder [PI : 

• I see no real benefit associated with non-compliance. 

QUESTION 7. 

What do you see to be the costs associated with non-compliance with this instruction? 
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Table A.7-1. Choices of Score Answers Offered to Stakeholders for Question 7. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 
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real 

costs 

costs 

cost 

costs 

cost 


Table A.7-2. Scores Provided by Stakeholders to Question # 7 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 
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K 

L 

M 

N 

O 
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Score 

5 
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5 

4 

4 
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4 

4 

4 

5 

5 

3 

1 

4 


Comments Provided by Stakeholders for Question 7: 

Stakeholder [A1 |TA,Im} : 

• I see many significant costs associated with non-compliance. 

• The proponent cannot perform adequately the needed design studies, trade 
studies, pre-test planning, pre-test predictions, and post-test analysis when the M&S is of 
poor quality, as would be indicated by its failure to be accredited. One cannot trust the 
results! Consequently, it is harder to analyze and solve problems early. This failure 
would be avoided if sufficient funding were applied to M&S development and its 
supporting VV&A. While the costs associated with non-compliance are sometimes hard 
to quantify, the effects are nonetheless apparent. 

Stakeholder FBI : 

• I see many significant costs associated with non-compliance. 

• M&S that does not meet the needs of the Tester / Accreditor does not 
produce useful M&S, i.e., it does not provide any benefit. Non-useful M&S would bring 
the program to a halt. 

Stakeholder [Cl : 

• I see many significant costs associated with non-compliance. 

• The many significant costs of non-compliance relate to the ultimate costs 
of‘doing it wrong’. 

Stakeholder FD1 : 

• I see several significant costs associated with non-compliance. 

• The costs associated with non-compliance depend on the importance of 
the model presented to the Tester / Accreditor. It seems to be anomalous for the 
proponent to develop a model to satisfy an OT requirement but not to comply with the 
VV&A process needed to convince the Tester / Accreditor. 
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Stakeholder [El : 

• I see several significant costs associated with non-compliance. 

• The costs would be very significant as they would be equal to the entire 
cost to develop and document the M&S, since the M&S would be rejected in its entirety 
for not proving to be fit for use under OT&E. 

Stakeholder FG1 : 

• I see one significant cost associated with non-compliance. 

• In the case of one ship acquisition program, it cost $250M (which had to 
be taken from other program funds) to perform the physical operational test that the M&S 
sought to accomplish in a virtual environment. Perhaps earlier PM interaction with the 
Tester / Accreditor and the collection of some additional physical (real) data would have 
led to a credible M&S solution. 

Stakeholder [HI : 

• I see several significant costs associated with non-compliance. 

• COTF holds the Program Managers to this instruction. Non-compliance 
could cause significant change in the acquisition program’s schedule (and thereby 
increase program costs). Thus the program’s assets would be tied up and would need to 
be rescheduled. 

Stakeholder [II : 

• I see several significant costs associated with non-compliance. 

• Several significant costs result from the various missed opportunities that 
non-compliance puts off, including: 

— Missed opportunities to consider early, cost-effective tradeoffs 
between M&S, T&E, and construction techniques. 

— Missed opportunities to develop complementary, cost-effective 
strategies to conduct T&E and M&S to achieve the objectives of T&E. 
M&S (and its VV&A) cannot be merely thought of as cheap way out 
of testing. 

Stakeholder [Kl : 

• I see several significant costs associated with non-compliance. 

• The Tester / Accreditor could fail parts of the PM’s system if he does not 
comply with their requirements for VV&A because they would not have the necessary 
confidence in the PM’s M&S. 

Stakeholder [LI : 

• I see many significant costs associated with non-compliance. 
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• There are lots of process costs associated with non-compliance, including 
the loss of engineering rigor, accountability, and repeatability. With non-compliance, 
one sees different programs using different processes. 

• The financial costs of non-compliance are evident when additional 
funding is required to build up the PM’s knowledge base to the level that has been 
already attained within COTF. Through compliance, the PM can leverage the knowledge 
of the Tester / Accreditor with little additional cost to achieve a successful M&S. 

Stakeholder [Ml : 

• I see many significant costs associated with non-compliance. 

• The PM relies on the Tester / Accreditor to accredit the model. The 
funding invested in the M&S is wasted without accreditation (accomplished through the 
VV&A process). 

Stakeholder FN1 : 

• I see one significant cost associated with non-compliance. 

• Not following the VV&A instruction creates a significant probability of 
incurring added costs, especially when trying to validate and verily the models of 
federations. On what basis will PMs be confident that the ship systems they are 
delivering will perfonn the way they are predicting? Physical testing will have to be 
added to prove the VV&A case. This is necessary because no one can buy or accept a 
product of which they are unsure, especially when the stakes are high. The costs are 
ultimately borne by the warfighter. 

Stakeholder FOl : 

• I see no real cost associated with non-compliance. 

• There is no real cost associated with non-compliance if VV&A is not 
done. However, the cost to go back later and conduct VV&A will be much more 
expensive. 

Stakeholder [PI : 

• I see several significant costs associated with non-compliance. 

• Non-compliance with the instruction would continue to cost the U.S. Navy 
more money. 

QUESTION 8. 

What do you see to be the risks associated with non-compliance with this instruction? 
Comments Provided by Stakeholders for Question 8: 
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Table A.8-1. Choices of Score Answers Offered to Stakeholders for Question 8. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 

minor 

real 

risks 

risks 

risk 

risks 

risk 


Table A.8-2. Scores Provided by Stakeholders to Question # 8 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

4 

5 

4 

4 

3 

4 

4 

4 

5 

5 

5 

5 

4 


Stakeholder FA1 {TA,lm} : 

• I see many significant risks associated with non-compliance. 

• The proponent cannot perform adequately the needed design studies, trade 
studies, pre-test planning, pre-test predictions, and post-test analysis when the M&S is of 
poor quality, as would be indicated by its failure to be accredited. One cannot trust the 
results! Consequently, it is harder to analyze and solve problems early. This failure 
would be avoided if sufficient funding were applied to M&S development and its 
supporting VV&A. While the risks associated with non-compliance are sometimes hard 
to quantify, the effects are nonetheless apparent. 

Stakeholder FBI : 

• I see several significant risks associated with non-compliance. 

• The risks are that the requirements of the M&S will not be met and 
therefore the OT&E test requirements would not be fully satisfied. 

Stakeholder [Cl : 

• I see many significant risks associated with non-compliance. 

• Models should be a level of fidelity higher (i.e., more detailed than) what 
you need. This would lower the risk that a detail important to the ship system’s predicted 
operational performance would be somehow missed. 

Stakeholder FD1 : 

• I see several significant risks associated with non-compliance. 

• There are very significant risks associated with non-compliance. The 
USN still has to accomplish an adequate OT&E. 

Stakeholder [El : 

• I see several significant risks associated with non-compliance. 
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• There are significant risks with non-compliance because OT&E 
requirements have to be satisfied. 

Stakeholder TGI : 

• I see one significant risk associated with non-compliance. 

• One significant risk associated with non-compliance was recently made 
evident where a ship acquisition program was held up at MS B while resolving the issue 
of whether the use of M&S would satisfy one of the OT requirements. 

Stakeholder [HI : 

• I see several significant risks associated with non-compliance. 

• Several significant risks are associated with non-compliant VV&A, 
including the need to “re-synchronize” or “re-coordinate” the acquisition program’s 
“tracks” (i.e., the kind of activities / events identified in the TEMP). As this would 
translate to cost increases, the PM would have to go back to the Resource Sponsor for 
help. But the Resource Sponsor’s solution is usually to advise the PM to “take it out of 
hide” and reprogram existing funds to support the changes. 

Stakeholder [II : 

• I see several significant risks associated with non-compliance. 

• Several significant risks result from the various missed opportunities that 
non-compliances put off, including: 

— Missed opportunities to plan the M&S (and its VV&A) to be a 
continual process throughout the ship’s acquisition lifecycle, which 
would achieve complete success for use of the M&S. 

— Missed opportunities to make strategic, up-front investments in the 
M&S (and its VV&A) that would achieve complete success for the 
acquisition program. 

— Missed opportunities to make appropriate replications of the real world 
in the M&S. 

• The devil is in the details, which tend to be overlooked when PMs don’t 
fully comply with the prescribed process. 

Stakeholder [Kl : 

• I see several significant risks associated with non-compliance. 

• If the PM does not comply with the VV&A requirements, the Tester / 
Accreditor could disapprove critical performance aspects of the PM’s system since they 
would not have confidence in the predictive quality of the M&S. Poor quality M&S 
entails erroneous predictions of system performance, which could lead to poor decisions 
that adversely impact the system’s operators. 
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Stakeholder [LI : 

• I see many significant risks associated with non-compliance. 

• Getting “false-positives” are the risks associated with non-compliance. It 
can be presumed that the PMs are moving forward in good faith to conduct VV&A on 
their M&S. However, when they do not use the COTF prescribed process, they end up 
having to develop their own. This will result in unevenness of results across the PMs. 
Some PMs will be more stringent than COTF, but most will likely be less stringent. 
Those who are less stringent run the risk of getting false-positive results, that is, of 
qualifying an M&S that is really deficient. The effects of that false positive will be 
rolled-up into the system to which the M&S is applied. The resulting disconnect between 
expected performance and actual performance means “bad news” for the fleet. 

Stakeholder [Ml ; 

• I see many significant risks associated with non-compliance. 

• There is little difference between the many risks and the many costs 
associate with non-compliance. They really go hand-in-hand, as the risks imply costs and 
the costs reflect risks. Any distinctions between the two relate more to the observer’s 
vantage point. 

Stakeholder FN1 : 

• I see many significant risks associated with non-compliance. 

• Not following the VV&A instruction creates a significant probability of 
incurring added risks, especially when trying to validate and verify the models of 
federations. On what basis will PMs be confident that the ship systems they are 
delivering will perform the way they are predicting? Physical testing will have to be 
added to prove the VV&A case. This is necessary because no one can buy or accept a 
product of which they are unsure, especially when the stakes are high. The risks are 
ultimately borne by the warfighter. 

Stakeholder FOl : 

• I see many significant risks associated with non-compliance. 

• Non-compliance with the instruction entails that there will be no 
confidence in the M&S proposed for use. 

Stakeholder [PI : 

• I see several significant risks associated with non-compliance. 

• There are several significant risks associated with non-compliance; the 
foremost among them is the lack of credibility for those models developed without regard 
to compliance. 
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QUESTION 9. 

To what extent did the change from the superceded COMOPTEVFORINST 5000.1 (of 5 
September 1995) to the new COMOPTEVFORINST 5000.1 A (of 9 September 2004) 
improve or degrade the VV&A of M&S used for OT&E? 


Table A.9-1. Choices of Score Answers Offered to Stakeholders for Question 9. 


(5) 

(4) 

(3) 

(2) 

to 

significant 

minor 

no real 

minor 

significant 

improvements 

improvements 

improvement 

degradations 

degradations 


Table A.9-2. Scores Provided by Stakeholders to Question # 9 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

3 

5 

5 

5 

3 

5 

3 

3 

4 

4 

3 

4 

4 


Comments Provided by Stakeholders for Question 9: 

Stakeholder fAl {TA, Im} : 

• I believe no real improvement to VV&A occurred. 

• The signed version of the 5000.1 A instruction (which excludes much of 
the detailed VV&A guidance that is provided by the surface warfare VV&A guide) does 
an adequate job of specifying the “who” “what” “when” “where” and “why” of VV&A 
but tends to be general in nature. It is particularly lacking in the “how to” guidance that 
the PMs seem to need. A more thorough VV&A instruction that provides greater detail is 
needed - one which could be tailored to match the specific needs of each program - 
especially in today’s environment of frequent personnel turnover, both in government and 
industry. An enhanced instruction would promote a faster learning curve for the PM. 
When exceptions to the normal VV&A process need to be made (e.g., combining the 
SMB and the SCP), stakeholders can nevertheless follow the intent of the VV&A 
guidance, since it is well understood and the programs can still perform all the tasks that 
lead to high quality and less rework. The latest draft of NMSO’s VV&A handbook 
provides guidance on this and more. The OTDs do not have the time to study and 
become experts in the technical details of M&S, so NMSO’s VV&A handbook is useful 
for those who need to understand more about M&S. On the other hand, the surface 
warfare VV&A guide is detailed, but only in the bottom-line management needed to 
conduct the VV&A. 

Stakeholder fBl : 

• I believe no real improvement to VV&A occurred. 

• The old instruction had much more detail, with appendices and guides. 
The new instruction is not complete. It tends to be too broad. Though it is well written, 
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it does not tell the PM what products COTF expects to get. Thus, the new instruction 
does not provide practical guidance so much as a very good over-arching philosophy. To 
make up for the detail that the new instruction lacks, the OTDs developed a Surface Ship 
VV&A Guide that provides some of the specific “how to” information that had been 
explicit in the old instruction. (Thus, some of the historical, working experience of U.S. 
Navy surface ship OT&E M&S VV&A is preserved.) The Surface Ship VV&A Guide is 
an interim solution. A revision to the new instruction is forthcoming (which may be 
providing the needed details and specific guidance). 

Stakeholder [Cl : 

• I believe significant improvements to VV&A occurred. 

• Significant improvements to VV&A guidance are seen in the new 
instruction. This includes the insight that the participants (within the established 
accreditation process) should move from a focus on the fidelity of the model to the 
appropriateness of the model for its intended use. The level of resolution to be achieved 
for the model depends on the questions being asked. 

Stakeholder FD1 : 

• I believe significant improvements to VV&A occurred. 

• The COMOPTEVFORINST 5000.1 (of 5 September 1995) did not place 
great emphasis in using M&S for OT&E. But people’s desire to participate (i.e., to use 
M&S for OT&E) changed from 1995 to 2004, whereby COMOPTEVFOR INST 
5000.1 A (of 9 September 2004) was issued. The keen interest in using (where 
practicable) M&S for OT&E was expressed by the CNO, and supported by COTF. 

• The change from the superseded instruction to the new instruction 
unquestionably improved the VV&A of M&S used for OT&E. It provides a far better 
road map than before. 

Stakeholder [El : 

• I believe significant improvements to VV&A occurred. 

• Significant improvements to VV&A guidance are seen in the new 
instruction. With the old instruction, the PMs did not know what the requirements were. 

• The earlier the OTD is involved, the better the compliance. The new 
instruction better promotes this. 

Stakeholder FG1 : 

• I believe no real improvement to VV&A occurred. 

• No real improvement to VV&A occurred in moving to the new 
instruction. It is not clear that the change to the new instruction would have improved the 
dynamic observed for one particular surface ship VV&A effort, where M&S was 
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developed to address a missile air defense requirement, such that live-fire testing could be 
avoided for events within a certain range of the self-defense test ship. Whether invoking 
the old or new instruction, the key issue would be the PM’s early interaction with COTF 
so that the credibility of the model could be established in time to avoid reprogramming 
funds and possible delays in the delivery of the system. 

Stakeholder [HI : 

• I believe significant improvements to VV&A occurred. 

• The changes incorporated in the new instruction are “golden,” as they 
allow the PM more flexibility while still expecting him to interact with COTF early in the 
process for delivering VV&A products. 

Stakeholder [II : 

• I believe no real improvement to VV&A occurred. 

• The new instruction kind of “hamstrings” the PM. On the one hand, it 
more clearly articulates COTF VV&A policy; but on the other hand, it does not provide 
the specific expertise (guidance) the PM needs to be able to implement the VV&A. 
While PMs can obtain the necessary practical guidance from other authorities (e.g., 
NMSO), it is important that COTF (as the customer) be satisfied with the process and 
products it receives. [Thus, the customer’s practical expectations should be better 
expressed in a revision to the new instruction.] 

• Regardless of making VV&A instruction changes, real improvement to 
VV&A would be achieved if PMs were to place more emphasis on addressing the 
technical challenges of M&S, which include: 

— Reducing model setup time. (This can be accomplished by CAD 
conversion for ship models.) 

— Running high fidelity models in real-time. 

— End-to-end Family of Systems (FOS) testing. (An example would be 
the “Testing in a Joint Environment Initiative.”) 

— Enterprise M&S efforts. 

— “No degrade” regression testing. 

— Emphasizing a “physics based” approach to VV&A. (Specific areas 
are: surrogate vice threat testing; measuring and modeling independent 
variables; contractor (tactical) software testing; and end-to-end 
testing.) 

— Recognizing where physics does not provide the basis for the VV&A. 
(Examples are: “Netcentric Warfare Knowledge Sharing” and “Social 
Network Performance and Measurement.”) 


Stakeholder [Kl : 

• I believe no real improvement to VV&A occurred. 


145 



• No real improvement to VV&A seems to have occurred in the move to the 
new instruction. It is difficult to detect any change in VV&A process outcomes without 
performing a detailed assessment, which requires time. 

Stakeholder [LI ; 

• I believe minor improvements to VV&A occurred. 

• Minor improvements to VV&A occurred in the move to the new 
instruction. For one, the new instruction provides a sense of structure, fonnat, and 
process. Combined with the ready advice and assistance of the OTDs at COTF, the new 
instruction guides the PM through the steps of the VV&A process with only minor 
difficulties. 

Stakeholder [Ml : 

• I believe minor improvements to VV&A occurred. 

• The new instruction better articulates accepted VV&A standards and 
provides more specific guidance than the general philosophy of the old instruction. 

Stakeholder FN1 : 

• I believe no real improvement to VV&A occurred. 

• No real improvement to VV&A occurred in the move to the new 
instruction (but it would be difficult to see any change as the new instruction has not been 
in effect that long.). 

Stakeholder jOj : 

• I believe minor improvements to VV&A occurred. 

• The new instruction is more user-friendly and shorter. 

Stakeholder [PI : 

• I believe minor improvements to VV&A occurred. 

• Some minor improvements seem to have resulted from the move to the 
new instruction (but it is difficult to specifically identify them). 

QUESTION 10. 

What improvements would you recommend for COMOPTEVFOR’s Accreditation 
process (beyond that prescribed by the new COMOPTEVFORINST 5000.1 A)? 

Comments Provided by Stakeholders for Question 10: 
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Table A.10-1. Choices of Score Answers Offered to Stakeholders for Question 10. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a few 

no 

significant 

significant 

significant 

minor 

real 

improvements 

improvements 

improvement 

improvements 

improvement 


Table A.10-2. Scores Provided by Stakeholders to Question # 10 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

4 

5 

1 

2 

4 

3 

1 

5 

3 

4 

2 

1 

4 

1 


Stakeholder FA1 {TA,Im} : 

• I recommend several significant improvements for COMOPTEVFOR's 
Accreditation process. 

• There are several significant improvements that can be made to 
COMOPTEVFOR's Accreditation process. Among these are: 

— Incorporate the “how to” guidance of the Surface Warfare VV&A 
Guide in to the COTF VV&A instruction. 

— Make the language more understandable to the average (and 
sometimes novice) user. 

— Make sure the process diagram and the mock schedule from the 
Surface Warfare VV&A Guide are included in the COTF VV&A 
instruction because these two diagrams provide a grand summary of 
the entire process. Instruction users could readily refer to these two 
pages and quickly formulate plans regarding requirements tasking, 
organization, planning and reporting documents, assessment and 
accreditation requirements, as well as the timing (schedule) required to 
perform all these tasks. 

— Incorporate the format and content examples that are currently 
provided at the end of the Surface Warfare VV&A Guide. 

• Since OT is the final step before full production, it would be most efficient 
to offer the PM a VV&A instruction that contains both policy and detailed “how to” 
guidance a single, usable resource. The PM would there find efficient methods to 
organize, estimate costs, determine schedules, and finish a quality M&S on time and 
within budget. 

Stakeholder FBI : 

• I recommend many significant improvements for COMOPTEVFOR's 
Accreditation process. 
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• There are several significant improvements that can be made to 
COMOPTEVFOR's Accreditation process. Among these are: 

— An appendix should be added to the instruction that would provide 
detailed examples of how to conduct VV&A. The appendix should 
provide the new PM a step-by-step model of how to go through the 
VV&A process. Such a guide would enable PMs to better comply. 

— Incorporate the VV&A tools provided by the Surface Warfare VV&A 
Guide, as this would increase the effectiveness of PMs. 

— Review the forthcoming new SECNAV VV&A instruction and derive 
/ incorporate the practical guidance that would apply to the VV&A of 
M&S used for OT&E into the COTF VV&A instruction. 

— Place greater emphasis on Tester / Accreditor participation on the 
SCP, as this would foster the early interactions needed to achieve 
compliance. 

— Review when and where (e.g., milestone decisions and review boards) 
VV&A is required and approved. Avoid imposing overly burdensome 
requirements. 

— Adequately plan for and fund the personnel needed to review the 
documentation produced by the SCP and other points in the VV&A 
process. 

— Provide a real “hammer” for the VV&A to be done. 

— Increase “ground-level” involvement in M&S and its VV&A. 

— Make the commitment to establish a quality approach to the VV&A 
effort. Program the needed funding and hire / employ qualified, smart, 
and disciplined personnel to accomplish the job. 


Stakeholder [Cl : 

• I recommend no real improvement for COMOPTEVFOR's Accreditation 
process. 

• There are no real improvements that can be made to COTF’s 
Accreditation process. The key issue for improving the accreditation process regards the 
implementation of VV&A and not necessarily what the VV&A requires by way of 
procedure. Instead of focusing on the process for producing documentation, the 
participants should focus on the technical content conveyed in the documentation. In 
other words, the instruction should not emphasize how to do conduct the VV&A process 
or require close adherence to a checklist. Rather, the focus should be on the 
appropriateness of the model to answer the technical question related to its intended use. 
In sum, the most prominent “content” issues for consideration are: 

— VV&A participants should move from a focus on the fidelity of the 
model to the appropriateness of the model for its intended use. 

— Models should be developed at one level of fidelity more detailed than 
what the proponent would otherwise need. 

— VV&A participants should improve their formal understanding of the 
model’s intended use. 
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Stakeholder [D1 : 

• I recommend a few minor improvements for COMOPTEVFOR's 

Accreditation process. 

• There are a few minor improvements that can be made for 

COMOPTEVFOR's Accreditation process. Upon initial review, there is nothing to 
reconsider regarding the new instruction. In moving from the old, it succeeded in 
capturing a new, needed perspective. Now, it should be noted that the new instruction is 
a philosophically related instruction. The forthcoming SECNAV VV&A Handbook is 
expected to provide more guidance on how to conduct VV&A. 

Stakeholder [El : 

• I recommend a few minor improvements for COMOPTEVFOR's 

Accreditation process. 

• Among the possible improvements to be made is finding a way to make 
models more available for reuse - both within the USN and among all the Military 
Services. Testing in the Joint Force Environment relies on a having an adequately 
modeled virtual environment. More funding is needed to fully support this initiative, 
along with a technical commitment at and above any particular Service’s ability. Then 
the question can be asked, “What does a PM expect to be able to do when he wants to 
reuse the M&S developed by another, only to find that this PM is not following the 
Service’s VV&A process?” 

Stakeholder FG1 : 

• I recommend one significant improvement for COMOPTEVFOR's 

Accreditation process. 

• Perhaps another revision of the instruction could be crafted to help the PM 
understand more clearly that unplanned funding requirements could be avoided if earlier 
interactions with the Tester / Accreditor are sought under the VV&A process. 

Stakeholder [HI : 

• I recommend no real improvement for COMOPTEVFOR's Accreditation 
process. 

• There is not enough information to see whether any real improvement 
could be made by another revision. 

Stakeholder [II : 

• I recommend many significant improvements for COMOPTEVFOR's 
Accreditation process. 
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• There are a number of important “lessons learned” from recent VV&A 
experience that can be emphasized or incorporated as guidance (etc.) in an improved 
COTF accreditation process, as follows: 

— M&S is not just a “tool” but a process. The products are an outcome 
of the process. (That M&S is commonly viewed as a tool (only) for 
acquisition programs to satisfy T&E requirements explains why M&S 
reuse is less than 10%.) The goal of M&S is to facilitate good 
acquisition. The objective of VV&A is to produce useful M&S. 

— M&S and VV&A activities have to be integral part of program and test 
process from inception. The intended purpose of the M&S must be 
clear. 

— M&S must have at least a CMMI level 3 process maturity to avoid 
VV&A problems. 

— Tradeoffs between M&S, T&E, and construction techniques should be 
considered early on. 

— M&S and VV&A must be a continual process throughout an 
acquisition lifecycle to be successful. 

— M&S and VV&A cannot be thought of as cheap way out of testing. 

— M&S requires up front investment to achieve success. 

• The bottom line is that the M&S proponent should do what is technically 
right. Properly perfonned VV&A will show this to all stakeholders. 

Stakeholder [Kl : 

• I recommend one significant improvement for COMOPTEVFOR's 
Accreditation process. 

• The practical application of any process is a difficult thing to address, no 
less improve. That being said, it seems that the new instruction can provide more 
guidance to the PM on the practical application of the VV&A process. 

Stakeholder [LI : 

• I recommend several significant improvements for COMOPTEVFOR's 
Accreditation process. 

• Of the several significant improvements that can be made to 
COMOPTEVFOR's Accreditation process, one that would be innovative to the PM is the 
deployment of an on-line process. Were such a capability implemented via a distributed 
knowledge base, it would allow for VV&A participants to work the process without the 
cost of traveling to remote sites for meetings. 

• Another improvement would be to implement “check sheets.” The current 
VV&A reporting requirements are way too complex. It seems that some activities see 
their participation in VV&A as a way to expend labor-hours and deliberately make the 
process complex as a means to keep them in business. Sometimes it appears that solving 
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the technical challenges of M&S is easier than managing the VV&A business process, 
with its myriad of meetings and reports. Some kind of reform is needed to simplify these 
complexities. 

• The role and function of NMSO with respect to supporting the PM’s 
efforts to comply with the instruction could be clarified in a revision to the COTF 
instruction. 

Stakeholder [Ml : 

• I recommend a few minor improvements for COMOPTEVFOR's 
Accreditation process. 

• Discussion related to model reuse should be incorporated in a future 
update of COMOPTEVFOR's VV&A instruction. This would be helpful in conveying 
the Tester / Accreditor’s standards and expectations for reuse. 

Stakeholder FN1 : 

• I recommend no real improvement for COMOPTEVFOR's Accreditation 
process. 

• No call for real improvement to COMOPTEVFOR's Accreditation process 
comes to mind. 

Stakeholder FOl : 

• I recommend several significant improvements for COMOPTEVFOR's 
Accreditation process. 

• There are several improvements that can be made to the COTF VV&A 
instruction, including the following items: 

— Reference the new Navy Standard, SND32. 

— Add process diagrams. 

— Provide more detail regarding M&S requirements and VV&A 
acceptability criteria. 

— Describe the expected relationship between the PMs and COTF (the 
Tester / Accreditor). 


Stakeholder [PI : 

• I recommend no real improvement for COMOPTEVFOR's Accreditation 
process. 

• COMOPTEVFOR's Accreditation process does not have need of 
improvement. The new instruction should be supported as is. Rather than look for any 
improvement in the COTF instruction, the U.S. Navy should focus its efforts on 
standardizing the process described by SECNAVINST 5200.40A. [Supporting this 
instruction would actually resolve the majority of compliance issues experienced with the 
COTF VV&A instruction.] Attention should also be made to the update and 
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implementation of NMSO’s VV&A Handbook. The resulting standardized process for 
VV&A is where proponents will see real cost benefits. 


OVERVIEW OF STAKEHOLDERS’S SCORES. 

Table A. 10-3, below, provides an overview of the scores provided by the 
stakeholders interviewed for questions #1 through #10. 


Table A.10-3. Overview of Stakeholders’ Scores 

Question: 

#1 

#2 

#3 

#4 

#5 

#6 

#7 

#8 

#9 

#10 

ID 

Stakeholder 

Group 

Sr. 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

A 

Tester/ 

Accreditor 

Lower 

5 

5 

5 

4 

3 

3 

5 

5 

3 

4 

B 

Tester/ 

Accreditor 

Lower 

5 

5 

5 

4 

1 

1 

5 

4 

3 

5 

C 

Tester/ 

Accreditor 

Middle 

5 

4 

5 

1 

2 

1 

5 

5 

5 

1 

D 

Tester/ 

Accreditor 

Upper 

5 

1 

4 

2 

1 

1 

4 

4 

5 

2 

E 

Tester/ 

Accreditor 

Upper 

5 

1 

5 

2 

2 

1 

4 

4 

5 

4 

G 

Resource 

Sponsor 

Middle 

1 

1 

4 

2 

2 

1 

3 

3 

3 

3 

H 

Resource 

Sponsor 

Lower 

2 

5 

4 

2 

2 

1 

4 

4 

5 

1 

I 

Resource 

Sponsor 

Upper 

2 

5 

5 

1 

1 

1 

4 

4 

3 

5 

K 

Program 

Manager 

Lower 

4 

4 

3 

2 

3 

1 

4 

4 

3 

3 

L 

Program 

Manager 

Middle 

4 

5 

5 

3 

1 

1 

5 

5 

4 

4 

M 

Program 

Manager 

Lower 

4 

4 

4 

1 

1 

2 

5 

5 

4 

2 

N 

Program 

Manager 

Middle 

2 

4 

5 

4 

2 

1 

3 

5 

3 

1 

O 

Program 

Manager 

Upper 

5 

5 

5 

4 

2 

1 

1 

5 

4 

4 

P 

VV&A 

Standards 

Upper 

4 

5 

4 

1 

1 

1 

4 

4 

4 

1 
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